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FOREWORD 


The  technical  directors  of  Navy  Laboratories  receive  Independent  Research  (IR)  and 
Independent  Exploratory  Development  (IED)  funding  annually  to  support  innovative, 
promising  work  without  going  through  the  formal  approval  procedures  required  under 
normal  funding  authorization.  This  funding  enables  the  researchers  selected  for  the 
program  to  devote  a  portion  of  their  time  to  basic  research  that  offers  potential  benefits 
for  specific  military  problem  areas. 

Research  at  the  Navy  Personnel  Research  and  Development  Center  addresses  the 
Navy's  needs  for  enhancing  system  and  personnel  performance  through  the  integration  of 
people  and  technology.  Resources  provided  for  the  IR/IED  program  have  been  used  to 
develop  a  variety  of  research  methods,  models,  and  techniques  within  the  areas  of 
training,  manpower  utilization,  organizational  productivity,  and  human  factors 
engineering  of  naval  weapon  systems  and  platforms. 

The  IR  program  has  been  active  at  this  Center  since  1973  and  is  funded  under 
Program  Element  (PE)  61 152N.  The  IED  program  was  initiated  in  1976  and  is  funded  by 
PE  62766N. 

This  report  is  submitted  to  fulfill  the  requirement  for  an  annual  IR/IED  report.  It 
provides  summaries  of  selected  FY85  projects,  the  program  funding  profile,  a  list  of  the 
honors  and  awards  IR/IED  researchers  earned  during  FY85,  and  a  list  of  FY85  publications 
and  presentations  resulting  from  IR/IED  efforts. 
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MODELS  FOR  CALIBRATING  MULTIPLE-CHOICE  ITEMS ' 

James  Bradford  Sympson 


Dichotomous  (right!  wrong)  scoring  of  multiple -choice  test  questions  does 
not  distinguish  among  various  wrong  answers  chosen  by  examinees.  Wrong 
answers  can  supply  valuable  information  about  examinee  ability.  A  new 
item-response  model  and  a  polychotomous  item-scoring  procedure  were 
developed  which  have  several  benefits  for  selecting  and  classifying  military 
personnel. 


BACKGROUND 

Mental  Testing.  For  more  than  40 
years,  the  selection  and  classification  of 
enlisted  military  personnel  has  depended 
heavily  on  objective  tests  of  mental 
ability.  These  tests  are  especially 
appropriate  for  selecting  and  classifying 
individuals  who  lack  specialized  training 
or  experience  and  must  undergo  formal 
training  in  preparation  for  their  job 
assignment. 

The  benefits  of  using  mental  ability 
tests  for  screening  and  classifying 
military  recruits  go  far  beyond  identify¬ 
ing  individuals  who  are  likely  to  perform 
well  during  their  initial  enlistment. 

Since  the  armed  services  promote  from 
within,  the  quality  of  personnel  available 
for  the  upper  enlisted  ranks  ultimately 
depends  on  the  quality  of  personnel 
accepted  for  initial  entry.  Thus,  both 
short-  and  long-term  outcomes  rely 
heavily  on  decisions  that  are  made  at 
initial  recruitment.  Mental  ability  tests 
increase  the  likelihood  that  these 
decisions  will  be  good  decisions. 

Multiple-choice  Questions.  Paper- 
and-pencil  mental  ability  tests,  such  as 
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the  Armed  Services  Vocational  Aptitude 
Battery  (ASVAB),  typically  consist  of 
multiple-choice  questions.  Even  though 
computerized  testing  will  probably  reduce 
the  use  of  paper-and-pencil  tests,  multiple- 
choice  items  will  continue  to  be  widely 
used.  Even  when  the  examinee  is  asked  to 
enter  a  "free  response"  on  a  computer 
(which  requires  the  examinee  to  recall 
rather  than  recognize  the  correct  answer), 
the  computer  must  assign  the  response  to 
one  of  several  predefined,  mutually- 
exclusive  categories.  Thus,  even  test 
questions  that  are  not  presented  in  a 
multiple-choice  format  will  usually  be 
scored  as  if  they  were  multiple-choice 
items. 


PROBLEM 

In  current  applications  of  multiple- 
choice  ability  testing  (e.g.,  the  ASVAB  and 
computerized  testing),  examinee  responses 
are  scored  as  either  correct  or  incorrect. 
This  dichotomous  item  scoring  procedure 
ignores  distinctions  among  the  various 
incorrect  answers  and  loses  information 
about  examinee  ability  that  could  be 
extracted  from  the  wrong  answers. 
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Mental  ability  te»ta  are  u»ed  as  an  aid  for  placing  recruits  Into  Navy  )obs. 


OBJECTIVE 

The  objective  of  this  research  is  to 
develop  psychological  measurement 
(psychometric)  procedures  to  extract 
information  from  examinees'  wrong 
answers  to  test  questions.  In  order  to 
accomplish  this,  we  are  developing 
polychotomous  item-response  models, 
new  testing  strategies  based  on  these 
models,  and  scoring  methods  that 
extract  information  about  examinee 
ability  from  all  item  responses— 
incorrect  as  well  as  correct. 


PROGRESS 

Several  polychotomous  item-response 
models  have  been  developed  and  tried  on 
actual  test  data.  The  most  promising  of 
these  models  is  being  evaluated  thoroughly. 
A  computer  program  that  computes 
response-option  scoring  weights  has  been 
developed  and  applied  to  several  sets  of 
test  data.  A  new  family  of  statistical 
distribution  functions  and  a  computer 
program  that  fits  this  distribution  function 
to  sets  of  test  scores  have  also  been 
developed. 


EXAMPLE 


A  25-item  test  of  arithmetic 
reasoning  ability  taken  by  1300  USMC 
recruits  was  analyzed  using  the  option- 
scoring  program  developed  in  this 
research.  Figure  1  shows  the  difference 
between  dichotomous  and  polychotomous 
item  scoring  for  the  following  multiple- 
choice  item  from  this  test: 

A  key  rack  has  8  rows  of  hooks. 
Each  row  has  6  hooks.  If  25  percent 
of  the  hooks  have  keys  on  them,  how 
many  hooks  are  empty? 

(a)  12 

(b)  16 

(c)  32 

(d)  36  (correct  answer) 


Sol\  .ig  this  problem  requires  three  steps: 

1 .  8  x  6  =  48  hooks. 

2.  48  x  .25  =  12  hooks  with  keys. 

3.  48  -  12  =  36  empty  hooks. 

Of  the  Marine  recruits  tested,  33  percent 
selected  option  "a”  as  the  correct  answer. 
Apparently,  these  individuals  completed 
the  first  two  steps  in  the  solution,  and  then 
stopped.  Although  option  "a"  is  incorrect, 
choosing  this  option  clearly  indicates  a 
higher  level  of  ability  than  does  choosing 
either  option  "b"  or  "c,"  which  are 
unrelated  to  the  sequence  of  steps  involved 
in  arriving  at  the  correct  answer. 

The  upper  portion  of  Figure  1  shows 
the  result  of  scoring  this  4-choice  item 
dichotomously.  Examinees  who  selected 
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Figure  I.  Models  lor  calibrating  multiple-choice  Items. 
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the  correct  answer  (33%)  were  assigned 
a  positive  ability-level  estimate,  while 
examinees  who  answered  incorrectly 
(67%)  were  assigned  a  negative  ability- 
level  estimate. 


The  lower  portion  of  Figure  1  shows 
the  result  of  scoring  this  same  item 
polychotomously.  Examinees  who 
answered  correctly  were  assigned  the 
same  ability  estimate  as  before,  but 
examinees  who  answered  incorrectly 
were  assigned  one  of  three  ability 
estimates  depending  on  which  incorrect 
answer  they  chose.  In  particular, 
examinees  who  selected  response  option 
"a"  received  a  positive  ability  estimate 
that  is  lower  than  the  one  assigned  to 
examinees  who  answered  correctly. 
Sorting  people  who  answer  incorrectly 
into  different  groups  provides  additional 
information  about  their  mental  ability 
and  increases  test  reliability. 

This  example  demonstrates  how 
additional  information  about  an 
examinee's  ability-level  can  be  extracted 
by  considering  which  incorrect  answer 
the  examinee  selected.  Treating  all 
wrong  answers  the  same  can  be  unfair  to 
those  examinees  who  have  given 
"partially  correct"  answers. 


The  importance  of  option  "a"  in  this 
item  was  discovered  using  the  psycho¬ 
metric  procedures  developed  in  this 
research.  These  procedures  are  based  on 
statistical  analyses  of  item  responses. 


They  do  not  require  reading  each  question 
to  discover  the  relationship  between 
ability  and  each  v/rong  answers. 

PLANS 

During  FY86,  we  will  apply  our 
polychotomous  item-response  model  to  a 
variety  of  test  questions.  We  will  also 
conduct  computer  simulations  to 
determine  the  model's  applicability  to 
computerized  adaptive  testing  (CAT). 


RESULTS 

Initial  results  indicate  that 
application  of  polychotomous  item- 
response  models  will  allow  us  to  shorten 
mental  ability  tests  by  about  13  percent 
without  sacrificing  test  reliability. 
Currently-used  dichotomous  item- 
response  models  fail  to  "fit"  a  significant 
portion  of  the  multiple-choice  items 
available,  whereas  our  polychotomous 
model  has  fit  every  test  item  to  which  it 
was  applied.  Thus,  more  of  the  test 
questions  that  have  been  written  can  be 
used  when  the  new  polychotomous  model 
is  applied.  Finally,  our  new  procedures 
enable  test  developers  to  identify 
questions  and  response  alternatives  that 
are  especially  good  or  especially  poor 
indicators  of  mental  ability  and  indicate 
to  the  test  developer  which  response 
alternatives  should  be  modified  to 
improve  item  performance.  These 
benefits  will  improve  the  measurement  of 
mental  ability  in  military  selection  and 
classification  tests. 


ENHANCING  UNDERSTANDING  OF  ELECTRIC  CIRCUITS 


William  E.  Montague 


The  Navy's  basic  electricity  and  electronics  (BE/E)  course  continues  to 
have  a  high  attrition  rate  despite  numerous  changes  in  course  content 
Qualitative  tests  developed  to  diagnose  student  problems  indicate  that 
students  do  not  understand  the  basic  laws  and  concepts  of  electricity  that  they 
need  to  understand  to  maintain  electronic  devices.  Current  tests  at  BE/E 
school  have  not  detected  this  serious  learning  deficiency.  The  results  suggest 
changes  in  training  materials  and  tests  to  enhance  student  learning.  Work 
with  the  training  command  to  improve  BE/E  instruction  will  continue. 


PROBLEM 

"Basic  electricity"  is  taught  as  a 
topic  in  science  courses  or  preparatory 
to  technical  courses  in  civilian  schools 
and  colleges,  vocational  programs,  and 
military  training.  The  Navy  teaches  a 
course  in  basic  electricity  and  elec¬ 
tronics  (BE/E)  to  about  25,000  trainee 
technicians  each  year.  Currently,  the 
course  provides  the  basic  "core" 
knowledge  and  general  test  equipment 
skills  that  trainees  need  before  they 
attend  about  20  different  follow-on 
courses  in  Navy  "A"  schools.  The  other 
services  provide  similar  large-scale  core 
training  for  trainee  technicians.  The 
content  and  sequence  of  topics  in  these 
courses,  secondary  school  and  college 
science  courses,  and  vocational  courses, 
are  very  similar.  These  courses  teach 
the  concepts  of  energy  potential, 
circuits,  resistance,  current,  and  power, 
which  students  must  understand  before 
they  can  understand  electronic  devices 
and  how  to  maintain  them. 


Research  studies  have  examined  the 
role  of  students'  cognitive  knowledge 
structures  in  learning  and  understanding 
the  physical  phenomena,  the  adequacy  of 
students'  scientific  concepts,  how  well 
students  explain  how  devices  work,  and 
why  devices  malfunction.  Universally, 
students  find  the  concepts  difficult  to 
learn  and  have  the  same  types  of  misunder¬ 
standing  during  learning.  Students  find  the 
physical-quantitative  models  difficult  to 
generalize  into  an  understanding  of  how 
devices  work  and  to  use  this  knowledge  to 
discover  reasons  for  malfunctioning. 
Emphasizing  the  quantitative,  physical 
science  explanations  of  the  underlying 
phenomena  does  not  seem  to  be  an 
effective  means  for  instructing. 

High  attrition  rates  have  plagued  the 
Navy's  course  and  follow-on  schools  have 
expressed  dissatisfaction  with  student 
knowledge  and  skills.  Despite  numerous 
attempts  over  the  last  decade  to  alleviate 
these  problems  by  revising  course  mate¬ 
rials,  the  attrition  rates  and  frequency  of 
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remediation  remain  about  the  same. 

I  his  fact  suggests  that  the  reasons  for 
the  problems  are  more  fundamental  to 
student  learning  and  understanding  than 
to  course  materials. 


The  present  research  was  under¬ 
taken  to  develop  a  means  of  detecting 
student  misunderstanding  and  to  examine 
ways  to  train  more  effectively.  Detailed 
analyses  of  student  learning  difficulties 
should  provide  a  basis  for  course 
revisions  based  on  the  principles  most 
appropriate  for  the  kinds  of  deficiencies 
revealed. 


BACKGROUND 

Work  during  FY83  and  FY8<* 
examined  the  BE/E  course  materials 
including  teaching  material  and  the  test 
items.  Data  on  the  errors  students  made 
on  test  items  were  obtained  to  deter¬ 
mine  whether  the  existing  tests  could  be 
used  to  diagnose  general  problems 
students  had  with  the  test  items.  The 
high  frequency  of  errors  students  made 
on  certain  module  tests  and  certain 
types  of  questions,  coupled  with  other 
reported  research,  suggests  that  students 
do  not  understand  the  relationship 
between  circuit  components  and 
functioning  that  the  basic  laws  and 
equations  of  electricity  describe.  The 
course  testing,  however,  is  relatively 
inadequate  for  describing  the  details  of 
student  misunderstanding.  Therefore, 
during  FY85,  we  developed  more 
systematic  means  for  identifying 
learning  difficulties  and  used  these 
techniques  to  determine  the  reasons  for 
student  errors  and  to  assess  the  effects 
of  changes  in  training  materials  on 
student  performance. 


Und*f»t«nding  b«»ir  electricity  corvrepts  is  essential 
lor  milntenonce  of  fleet  systems. 


OB3ECTIVES 

Project  goals  in  FY85  were  to  (1) 
identify  deficits  and  flaws  in  students' 
solutions  to  test  problems  about  simple 
circuits,  (2)  develop  a  summary  model  to 
describe  the  reasons  students  make  these 
errors,  and  (3)  test  the  effects  of  changes 
in  instructional  materials  on  student 
errors. 


APPROACH 

Prior  to  FY85,  diagnostic  tests  had 
been  developed  to  improve  the  means  for 
examining  student  errors  from  which  to 
infer  student  understanding  of  circuits. 
During  FY85,  these  tests  were  used  to 
evaluate  changes  in  BE/E  course  materials 
and  to  test  the  use  of  an  analogy  in 
reducing  errors.  The  diagnostic  tests 
differ  from  the  tests  used  in  the  current 
course  in  that  they  are  qualitative  rather 
than  quantitative.  This  eliminates 
calculation  errors,  w'hich  are  not  directly 
relevant  to  circuit  understanding  and 
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confuse  the  analysis.  In  addition, 
because  a  student  has  to  indicate  what 
happens  to  every  circuit  component  O'- 
measurement  as  a  result  of  some  change 
in  the  circuit,  patterns  of  errors  related 
to  misconceptions  (flawed  or  "buggy" 
concepts)  are  detectable.  Analysis  of 
these  test  data  concentrated  on 
replicating  the  error  patterns  Riley 
(1984)  found  in  an  analysis  of  student 
protocols  taken  while  students  studied 
the  BE/E  training  materials  and  was 
done  with  her  cooperation. 

During  FY85,  test  data  collected 
from  four  groups  of  students  in  FY84 
were  analyzed  to  identify  the  students' 
flawed  concepts.  Additionally,  48  Navy 
BE/E  students  using  current  course 
materials  (Coursefile  72)  and  46  students 
using  course  revised  materials 
intended  to  be  implemented  in  the  fall 
of  1985  took  the  diagnostic  tests.  Also, 
BE/E  materials  incorporating  an  analogy 
to  improve  the  development  of  student 
understanding  of  circuit  functioning 
usually  described  by  the  Ohm’s  Law 
equations  were  administered  to  five 
college  students  to  clarify  some  issues 
that  arose  in  prior  research. 


ACCOMPLISHMENTS 

The  data  obtained  during  FY85 
generally  agree  with  those  obtained  from 
the  less  extensive  studies  run  in  prior 
years.  In  general,  students  who  success¬ 
fully  completed  the  BE/E  modules 
covering  basic  direct  and  alternating 
current  circuits  have  substantial  flaws  in 
their  understanding  oi  how  simple  DC 
circuits  function.  Most  students  made 
multiple  errors  on  the  diagnostic  tests 
with  fewer  than  ten  percent  making  no 
errors.  Furthermore,  analysis  of  the 
errors  revealed  considerable  homoge¬ 
neity  in  the  types  of  errors  the  students 


made.  For  example,  80  percent  of  the 
errors  reveal  that  the  students  misunder¬ 
stand  Kirchoff's  Law  about  the  distribution 
of  voltage  in  circuits,  which  is  fundamental 
to  developing  troubleshooting  skills.  One 
wonders  why  current  tests  do  not  detect 
such  a  gross  problem  and  how  to  correct 
the  flaws. 

An  analysis  of  the  protocols  taken 
while  the  students  studied  materials 
containing  the  analogy  designed 
specifi  cally  to  assist  comprehension  of 
circuit  function  provides  some  ideas  that 
might  help.  This  analogy  (Riley,  1984) 
represents  a  voltage  as  a  stack  of  chips  or 
tokens  and  resistors,  as  boxes  divided  into 
slots  representing  resistance.  In  a  series 
circuit,  the  chips  are  distributed  equally  to 
all  slots.  The  number  of  tokens  per  slot 
i  Lprescnts  current.  This  model  provides  a 
concrete  representation  of  the  relation¬ 
ships  summarized  by  Ohm's  and  Kirchoff's 
Laws.  It  is  easy  for  students  to 
understand,  requires  no  manipulation  of 
abstract  equations,  and  seems  to  aid 
student  performance  during  learning. 
Students  using  this  analogy  made  fewer 
errors  than  did  students  using  the  standard 
course  materials.  These  results  suggest 
that  training  can  be  improved  by  providing 
readily  understood  models  that  make 
circuit  functioning  apparent. 

The  analysis  ol  the  data  from  the  two 
student  groups  (48  and  46  respectively)  is 
not  yet  complete.  The  group  using  the 
revised  materials  performed  slightly  better 
on  one  of  the  current  comprehensive  BE/E 
course  tests.  Data  obtained  from 
comparisons  of  test  scores  for  groups  using 
the  revised  material  at  different  locations 
revealed  statistically  reliable  (but  small) 
improvement  over  the  groups  using  the 
current  course  materials.  However,  on  the 
diagnostic  tests,  most  of  the  students  in 
both  groups  continued  to  make  substantial 
numbers  of  errors.  Although  analysis  of 
the  data  from  these  groups  is  incomplete, 


students'  errors  were  the  same  type  as 
before.  Whether  or  not  there  are 
declines  in  the  different  error  types  that 
can  be  attributed  to  the  revision  of  the 
materials  is  yet  to  be  determined. 


CONCLUSIONS  AND 
RECOMMENDATIONS 

Research  designed  to  analyze  error 
patterns  in  student  responses  to  circuit 
problems  reveals  the  extent  of  student 
misunderstanding  and  provides 
suggestions  both  for  practical  testing 
and  changing  training  materials  to 
enhance  student  learning  and,  hopefully, 
their  future  performance  on  the  job. 

The  difficulty  students  have  in  solving 
circuit  problems  seems  due  to  imperfect 
conceptual  or  mental  models  of  circuit 
functioning.  These  imperfections  can  be 
summarized  by  a  few  rules  and  then 
systematically  diagnosed  by  appropriate 
tests.  At  the  least,  diagnostic  testing  of 
this  kind  should  be  incorporated  into 
course  procedures.  The  reasons  that 
current  tests  do  not  indicate  the  serious 
deficiencies  existing  in  student  learning 
are  not  yet  definite.  However,  it  seems 
likely  that  students  concentrate  so  much 
on  solving  the  quantitative  equations, 
emphasized  by  current  teaching  and 
testing,  that  they  tend  to  ignore  circuit 
functioning.  This  idea  needs  to  be  tested 
in  future  work. 


APPLICATION 

The  BE/E  school  adopted  some  of 
the  qualitative  test  items  used  in  the 
diagnostic  test.  When  students  made 
many  errors  on  these  items,  instructors 
examined  and  revised  the  corresponding 
instructional  materials.  Although  our 


evaluation  of  these  revisions  is  incomplete, 
they  may  have  caused  the  slight  improve¬ 
ment  performance  noted  on  the  course 
tests.  These  course  materials  will  be  used 
in  all  BE/E  schools  starting  in  November 
1985.  The  training  command  has  requested 
that  we  evaluate  the  new  materials  and 
compare  the  effectiveness  of  the  current 
self-paced  delivery  system  with  a  lecture- 
based  one,  beginning  in  April  1986.  The 
diagnostic  tests  developed  here  will  be 
used  in  these  assessments. 


PLANS 

Plans  for  the  coming  year  include 
continuing  the  analysis  of  the  existing 
data.  Further  studies  of  the  effects  of  the 
structure  of  instructional  materials  on 
student  understanding  will  be  undertaken. 
We  will  continue  to  develop  and  test  ways 
of  identifying  the  problems  students  have 
in  learning  basic  electricity  and  assist  with 
the  evaluation  of  new  course  materials. 
Methods  are  being  developed  to  examine 
the  students'  conceptions  of  current, 
circuit,  power,  etc.  Students'  capability 
for  working  with  equations  and  in 
qualitative  reasoning,  which  are  assumed  in 
the  course,  will  be  examined.  Course 
materials  will  be  prepared  specifically  to 
overcome  student  misunderstanding 
identified  by  the  diagnostic  tests  and  their 
effectiveness  in  reducing  or  eliminating 
these  learning  flaws  will  be  tested. 
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COGNITIVE  AND  EMOTIONAL  PROCESSING 


Barbara  A.  McDonald 


A  need  for  the  Navy  to  understand  cognitive  processing  exists  because  of 
applications  to  training.  Usually,  the  influence  of  emotion  on  cognition  is  not 
addressed  even  though  many  Navy  tasks  are  performed  under  emotion- 
producing  conditions.  A  model  is  presented  which  does  include  emotion  and 
characteristics  of  the  individual  in  cognitive  processing.  It  promises  a  better 
understanding  of  individual  cognition. 


INTRODUCTION 


Models  of  human  cognition,  which 
influence  educational  research,  assume 
that  information  processing  is  continuous 
and  rational  whereas  emotional  proc¬ 
essing  is  intermittent,  and  if  anything, 
exerts  an  irrational  effect  on  cognition. 
Both  cognitive  and  emotional  processes 
are  continuous  and  simultaneous,  and  the 
assumption  of  rationality  for  either  is 
misguided.  Further,  although  cognitive 
and  emotional  processes  may  be  separate 
processes,  they  arc  interactive.  In  fact, 
it  will  be  shown  that  emotion  is  needed 
to  interpret  cognitive  processes.  As  it 
happens,  everyday  experience  and 
experimental  evidence  suggests  that 
behavior  patterns  do  not  make  much 
sense  when  explained  by  totally 
cognitive  processes,  anyway.  A  model 
that  accounts  for  more  person-oriented 
impacts  on  information  processing  is 
needed.  The  purpose  of  this  paper  is  to 
discuss  work  on  such  a  model  at  the 
Navy  Personnel  Research  and  Develop¬ 
ment  Center. 


In  recent  attempts  to  relate 
emotion  to  cognition  (Bower,  1981) 
Izard,  1984;  Lazarus,  1984;  Mandler, 
1984;  Tompkins,  1981;  Zajonc,  1980), 
several  theorists  have  assumed 


that  emotion  affects  cognitive  processing 
only  after  first  being  appraised  by 
cognitive  processing.  The  issue  is  whether 
emotion  needs  to  be  recognized  by  the 
individual  before  it  can  influence  cognitive 
processing.  In  fact,  Lazarus  says  the 
question  is,  "How  cognition  shapes 
emotion."  Another  perspective  on  the 
interaction  between  cognition  and  emotion 
concerns  the  storage  of  emotion.  Bower, 
for  example,  suggests  that  each  specific 
emotion  is  stored  in  memory  as  a  node  with 
associative  pointers  to  its  many  aspects; 
thus,  emotion  is  stored  in  memory  the 
same  way  any  other  concept  is  stored.  The 
ways  phenomena  are  investigated  ranges 
from  looking  at  preferences  (Zajonc,  1980), 
to  memories  for  events,  to  storage  of 
emotionally  laden  memories.  With  the 
exception  of  Mandler's,  no  model  deals 
with  ongoing,  simultaneous  information  and 
emotional  processing.  Furthermore,  none 
has  emphasized  the  characteristics  of  the 
individual  and  how  they  might  mediate  the 
relationship  between  cognitive  and 
emotional  processing. 

The  model  developed  in  the  present 
research  differs  from  the  others.  It 
emphasizes  simultaneous  ongoing  cognitive 
and  emotional  processing,  as  opposed  to 
the  storage  of  emotionally  related 
inf  irmation  or  labeling  of  emotion.  The 
model  also  considers  the  person-related 
variables  critical,  i.e.,  the  person 
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who  is  doing  the  processing  is  important. 
Finally,  this  model  does  not  consider  a 
direct  effect  of  emotion  on  cognitive 
process,  but  rather  looks  at  an  indirect 
effect  mediated  by  individual  difference 
filters. 


A  MODEL  OF  COGNITIVE  AND 
EMOTIONAL  PROCESSING 

This  work  is  particularly  directed 
toward  explaining  simultaneous  cognitive 
and  emotional  processing  in  the  ongoing 
"here  and  now."  When  information  is 
processed  in  any  situation,  the  cognitive 
processes  such  as  attention,  perception, 
and  memory  operate  on  the  information 
so  it  can  be  understood  and  stored  for 


later  use.  The  model  to  be  described 
here  adds  to  the  cognitive  model  by 
including  characteristics  of  the  individual, 
and  the  emotions  of  the  individual,  and 
shows  how  they  play  an  important  part  in 
how  the  information  will  be  processed. 

F  igure  1  describes  the  relationship  between 
characteristics  of  the  individual,  emotional 
arousal,  and  cognitive  processing.  This 
diagram  shows  that  instead  of  affecting 
cognitive  processing  directly,  emotional 
arousal  affects  cognitive  processing 
indirectly  by  activating  the  appropriate 
individual  difference  "filters."  The 
individual  difference  filters  that  operate 
may  include,  for  example,  personality 
variables,  the  self-concept,  and  perceptual, 
cultural,  and  prior  experience  factors.  The 
present  work  focuses  on  personality,  self- 
concept,  and  prior  experience  as  filters. 


PERCEPTION 


Figure  1.  Processing  of  Information  depends  on  the  combi  nation  of  smotlonal  arousal 
and  Individual  difference  fitters. 
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ASSUMPTIONS  AND  COMPONENTS 
OF  THE  MODEL 

Both  emotional  and  cognitive  proc¬ 
esses  are  ongoing  within  an  individual  at 
all  times.  Whereas  cognitive  processes 
transform  information,  emotion  trans¬ 
forms  physical  and  chemical  energy  into 
autonomic  arousal.  In  fact,  separate 
neuroanatomical  structures  have  been 
postulated  for  cognitive  and  emotional 
processing.  The  limbic  system  has  been 
called  the  "emotional  brain"  and  the 
hypothalamus  has  been  implicated  in 
cognitive  processing  (Zajonc,  1980; 

Izard,  1984). 


The  knowledge  base  (the  specific 
information  being  processed)  and  how 
the  individual  difference  filters  work  are 
also  features  of  the  model.  First,  the 
model  assumes  that  when  it  comes  to 
processing  incoming  information,  the 
person  usually  processes  information 
from  past  experiences  as  well  as  the  new 
information  that  is  coming  in.  The 
extent  to  which  old  information  is 
processed,  as  opposed  to  the  "pure" 
cognitive  processing  of  new  information, 
depends  on  the  intensity  of  emotional 
arousal  at  the  time  and  how  much  the 
new  information  triggers  a  response 
from  the  filters.  The  filters  interact 
with  emotion  and  cognitive  processes  by 
generalizing  from  new  (incoming)  stimuli 
to  past  (experienced)  stimuli.  They  act 
to  make  the  person  see  problem 
elements  in  the  same  way  over  time,  to 
making  decisions  in  a  similar  way  over 
time,  and  to  narrow  perceptual 
attention,  for  example.  The  cognitive 
system  may  not  "know"  that  emotion  is 
operating  in  the  sense  of  having  labeled 
the  emotion.  People  often  distort  or 
deny  emotions  and  their  effects. 
Mechanisms  that  account  for  this  in 
personality  theory  are  called  defense 
mechanisms. 


The  individual  difference  filters 
interact  with  emotional  arousal,  cognitive 
processes,  and  the  information  being 
processed  at  the  moment  (knowledge  base). 
Emotional  arousal  determines  the  salience 
of  the  different  filters,  while  the  filters 
influence  the  level  of  emotional  intensity, 
especially  in  relationship  to  the  knowledge 
base  being  processed.  Some  information 
will  be  more  emotionally  laden  for  one 
person  than  another,  depending  on  past 
experience.  The  relationship  between  the 
filters  and  cognitive  process  is  an 
important  one  because  the  filters  have 
been  shaped  over  time  through  interaction 
with  cognitive  processes. 


In  the  model,  the  fact  that  filters  are 
necessary  to  influence  cognitive  process  is 
assumed  partly  because  emotions  do  not 
affect  individuals  in  the  same  way.  This 
has  often  been  dealt  with  by  postulating 
"individual  differences"— personal 
characteristics  that  are  measurable.  We 
are  taking  individual  differences  a  step 
further  in  suggesting  that  they  have 
functional  properties  that  influence 
cognitive  processing.  Filters  are 
particularly  interesting  because  they 
influence  cognitive  processes  to  a  greater 
or  lesser  extent  based  on  the  intensity  of 
emotional  arousal.  The  intensity  of 
emotion  aroused  in  combination  with  the 
filters  determine  the  aspects  of  the 
knowledge  base  that  will  be  salient. 


EXAMPLES  OF  FILTERS 

As  an  example,  consider  how  person¬ 
ality  and  self-concept  operate  as  a  set  of 
filters  to  cognitive  processing.  In  general, 
personality  characteristics  and  an 
individual's  self-concept  filter  cognitive 
processing  by  predisposing  a  person  to 
handle  information  in  certain  ways. 

Among  the  personality  characteristics 
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relevant  to  the  processing  of  information 
are  predisposition  to  levels  of  anxiety, 
defense  mechanisms,  and  coping 
strategies.  A  person  characterized  by 
high  anxiety  would  be  more  likely  to 
experience  a  narrowing  of  attention  and 
subsequent  "mental  blocks"  on  test 
items.  A  person  who  handled  anxiety  by 
getting  angry  would  perhaps  blame  the 
instructor  for  a  poor  grade  instead  of 
attempting  to  find  out  what  the  solution 
to  the  problem  was.  In  terms  of  self- 
concept,  people's  feelings  are  usually 
tied  to  their  self-concept  so  that  if  they 
feel  they  are  competent,  they  will  tend 
to  explore  more  hypotheses,  test  more 
solutions,  and  stay  at  problem  solving 
much  longer  than  a  person  who  feels 
self-doubt.  Thus,  self-concept  "filters" 
both  perception  of  the  task  ("I  can  do 
this  if  I  keep  trying")  and  the  processes 
called  up  to  perform  the  task. 


Prior  experience  operates  as  a  very 
effective  filter  because  familiarity  is  a 
big  factor  in  determining  how 
information  will  be  processed.  Prior 
experience  affects  what  will  be  attended 
to,  what  will  be  of  interest,  how 
comprehension  will  proceed,  what  will  be 
remembered,  and  how  information  will 
be  used  in  the  future.  Prior  experience 
helps  filter  information  according  to 
stereotypes  and  prejudices,  in  the  best 
and  worst  senses.  Experience  can  blind  a 
person  to  new  aspects  of  the  situation 
but  it  can  also  cut  short  the  laborious 
procedures  of  decision  making. 


HOW  THE  MODEL  WORKS 

In  most  information  processing 
experiences,  people  deal  with 
information  in  ways  that  are 
comfortable  for  them.  They  use  routines 
that  fit  what  they  expect  to  learn  and 
how  tney  have  learned  in  the  past.  As 
shown  in  Figure  cognitive  processing 
is  affected  by  emotions  as  they  influence 


an  individual's  filters.  People,  in  general, 
process  information  with  a  bent  toward 
what  they  are  interested  in,  what  they 
know  best,  what  threatens  them  the  least, 
what  they  are  culturally  familiar  with, 
what  they  need  to  know,  and  many  other 
things.  In  this  way,  emotions  can  influence 
cognition  by  filtering  the  use  of  cognitive 
processes  such  as  the  attentional, 
perceptual,  and  memory  systems.  Emotion 
does  not  affect  cognitive  processing 
directly  but  does  affect  the  salience  of  the 
individual  difference  filters  that  do.  This 
interaction  between  emotion  and  cognition 
happens  moment-by-moment  as  people 
process  information.  It  does  not  have  such 
a  big  effect  that  it  seriously  distorts  the 
cognitive  processes  in  most  situations;  that 
is,  the  information  being  processed  is 
usually  more  important  than  the  filters. 

The  role  of  filters  under  conditions  of  low 
arousal,  then,  is  to  automatize  cognitive 
processes,  making  shortcuts  of  many 
decision  points.  A  qualitative  shift  in  the 
processing  of  information  can  take  place, 
however,  when  strong  emotions  are 
aroused,  so  the  mutual  influence  of 
cognition  and  emotion  can  have  a  big 
impact.  It  is  at  this  point  that  highly 
filtered  cognitive  processing  takes  place. 

When  emotional  arousal  is  high  (either 
positive  or  negative),  its  influence  on 
cognitive  processing  can  be  extreme.  A 
reaction  (most  likely  a  physiological 
reaction)  within  the  person  changes  the 
way  in  which  information  is  processed.  At 
this  point,  there  is  extreme  selective 
enhancement  of  individual  difference 
filters.  Although  activation  will  not 
necessarily  occur  on  a  conscious  basis  when 
emotions  are  aroused,  an  individual  has  less 
energy  to  moderate  the  use  of  ail  the 
filters  that  generally  operate  and  the 
filters  start  determining  the  way  the 
information  will  be  dealt  with.  In  this 
case,  information  will  be  distorted  (even  if 
it  is  to  the  positive)  and  the  resulting 
reactions,  decisions,  opinions,  and 
memories  for  the  event  will  be  more 
extreme  and  reflect  more  of  the 
individual's  biases. 
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One  can,  therefore,  view  the 
dynamic  interplay  between  emotion  and 
the  filters  as  existing  on  a  continuum. 
With  low  arousal,  the  filters  are  present 
and  working,  but  they  are  fluid,  the 
effect  is  there  but  the  person  is  still 
mainly  processing  information.  With 
medium  level  arousal,  the  person  is 
attending  to  information  but  is  actively 
searching  for  coping  mechanisms  to  deal 
with  the  processing  demands  (either 
because  there  is  emotion  related  to  the 
specific  information  being  processed  or 
because  there  is  competing,  external 
emotion  unrelated  to  the  current 
processing  task).  At  a  high  level  of 
emotional  arousal,  the  filters  exert  the 
most  rigid  influence  over  cognitive 
processing. 

The  primary  features  of  this  model 
are  (1)  the  simultaneous,  ongoing  nature 
of  both  cognitive  and  emotional 
processing,  and  (2)  the  way  in  which  the 
individual-difference  filters  operate 
between  these  two  processes.  An 
experimental  assessment  was  therefore 
desired  to  allow  observation  of  these 
features. 


feasibility  of  the  experimental  scenario  as 
well  as  investigated  the  appropriateness  of 
the  suggested  model  of  simultaneous 
cognitive  and  emotional  processing. 


In  the  experiment,  seven  Navy 
electronic  warfare  specialists  who  were 
either  tactics  experts  or  novices  played  a 
computer-based  game  of  military  tactics. 
Three  conditions  were  used:  in  the  first 
session,  the  game  was  played  under  normal 
conditions;  in  the  second  session,  the  game 
was  emotion-producing;  in  the  third 
session,  normal  conditions  were  used  again. 
The  computer  collected  data  on  the 
accuracy  of  tactical  decisions,  the  skill 
strategies  used,  and  the  timing  of  strategy 
use.  All  three  sessions  were  tape-recorded 
to  capture  emotional  reactions.  The 
experimenter's  role  was  to  probe  the  player 
for  strategy  and  content  knowledge.  This 
experimental  situation  was  successful  in 
allowing  the  subjects  to  play  the  game 
while  being  observed.  Observed  were  score 
decrements,  strategy  differences,  timing 
changes,  and  changes  in  decisions  made.  In 
addition,  emotion  was  observed  and 
recorded. 


THE  STUDY 

The  experimental  method  was 
developed  to  meet  several  criteria:  (a) 
ongoing  behavior  needed  to  be  observed 
within  a  scenario  that  allowed 
simultaneous  collection  of  performance 
data  and  emotional  reactions,  (b)  similar 
conditions  had  to  be  present  for  all 
subjects,  (c)  the  task  needed  to  be 
observed  under  normal  as  well  as 
emotion-producing  conditions,  and  (d) 
the  task  had  to  be  realistic  and  absorbing 
for  the  participants. 

The  experimental  scenario 
developed  to  meet  these  criteria  was  a 
computer-based  game  of  military 
tactics.  A  pilot  study  tested  the 


Thlt  {•miliar  military  icene  cannot  be  underatood 
without  conalderlng  the  effects  of  emotional  arousal. 
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The  game  was  very  realistic  and 
produced  identifiable  emotion.  There 
were  differences  in  behavior  from 
session  1  to  session  2  (when  the 
conditions  of  the  experiment  changed) 
and  emotions  changed  quite  a  bit  under 
the  second  condition.  The  way  emotion 
was  expressed  was  also  different, 
depending  on  the  personality  and  seif- 
concept  of  each  individual. 

One  finding  which  was  very  sur¬ 
prising  to  the  experimenter  was  that  in 
the  conversations  that  ensued  in  the 
third  session,  the  return  to  normal 
gaming,  all  but  one  person  discussed 
more  personal  information  than  in  the 
first  session.  Conversation  centered 
around  job-related  experiences  which 
had  either  caused  them  great  pain  or  had 
resulted  in  failed  expectations.  An 
interpretation  for  this  finding  is  that  the 
stressful  game  condition  elicited 
reactions  similar  to  situations 
experienced  in  the  past.  In  general,  the 
men  seemed  to  feel  comfortable 
expressing  emotions  to  the  female 
experimenter. 


CONCLUSIONS 

The  results  of  this  study  are  pre¬ 
liminary.  Many  issues  still  need  to  be 
addressed  in  the  research  and  at  least 
one  other  group  must  be  assessed. 
Objective  measures  of  emotional  arousal 
will  be  collected  in  future  research. 
However,  the  findings  to  date  are  very 
encouraging. 

The  simultaneous  nature  of  both 
cognitive  and  emotional  processes  was 
observed  and  the  effects  of  emotional 
arousal  on  cognitive  process  as 
moderated  by  the  individual's  charac¬ 
teristics  was  observed  as  well.  Further 
work  to  assess  this  model  of  cognitive 
and  emotional  processing  is  planned. 
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COGNITIVE  FACTORS  IN  LEARNING  AND  RETENTION 

Paula  3.  Konoske 
3ohn  A.  Ellis 


Procedural  tasks,  which  consist  of  an  ordered  sequence  of  steps  performed 
on  a  single  object  or  in  a  specific  situation  (e.g.,  preventive  maintenance), 
comprise  the  majority  of  tasks  performed  in  the  military.  The  Navy 
instruction  for  these  tasks  is  usually  a  sequence  of  steps  with  one  goal  that 
requires  rote  memorization.  There  is  little  explanation  about  the  system  and 
how  the  steps  interrelate.  Since  procedural  skills  are  often  not  retained  well 
in  memory,  this  research  examined  methods  for  designing  instruction  that 
make  procedural  skills  more  resistant  to  forgetting.  Elaborated  instructions 
that  provided  a  how/why  explanation  prior  to  learning  a  procedural  task  were 
shown  to  facilitate  retention  of  that  skill.  The  results  are  relevant  to  the 
design  of  Navy  instruction  with  respect  to  how  much  "theory  "  is  presented 
and  the  effect  of  this  on  the  length  of  the  training  pipeline. 


PROBLEM 

Recent  surveys  have  found  that 
procedural  tasks  comprise  the  majority 
of  tasks  performed  in  the  military 
Fredericks,  1981;  Tarr,  1985;  Van 
Kekerix,  1983).  Further,  Campbell, 
O'Conner,  and  Peterson  (1976)  found  that 
procedural  tasks  are  the  most  important 
and  necessary  type  of  task  for  Navy 
mission  readiness.  Procedural  tasks 
consist  of  an  ordered  sequence  of  steps 
or  operations  performed  on  a  single 
object  or  in  a  specific  situation.  They 
involve  few  decision  points  and  are 
generally  performed  the  same  way  each 
time  the  task  is  done.  Procedural  tasks 
vary  in  (1)  the  amount  of  planning 
required  to  accomplish  the  task,  (2)  the 
number  of  steps  and  subprocedures 
required  to  accomplish  the  task,  (3)  the 
amount  of  cueing  built  into  the  task,  (4) 
the  number  of  decision  points,  (5) 
whether  or  not  the  order  the  steps  are 
performed  can  vary,  and  (6)  whether  the 
goals  of  the  task  are  internal  or  external 
to  the  task,  system,  or  situation  (e.g., 
operation  vs.  maintenance).  Military 
personnel  in  technical  ratings  must 


Model  crane  that  was  used  by  subjects 
lor  testing  procedural  task  retention. 


maintain  high  levels  of  procedural  skill 
and  knowledge  to  be  able  to  perform 
their  jobs  successfully.  Unfortunately, 
research  has  shown  that  procedural  tasks 
are  frequently  not  well  retained  (Ellis, 
1979;  Hurlock  3c  Montague,  1982; 
Schendel,  Shields,  3c  Katz,  1978; 
Vineberg,  1975;  ).  A  possible  solution  to 
this  retention  problem  and  the  focus  of 
this  research  is  to  develop  methods  for 
designing  instruction  that  make  pro¬ 
cedural  skills  more  resistant  to 
forgetting. 

This  paper  describes  briefly  (1)  the 
development  of  and  research  support  for 
a  taxonomy  of  qualitative  explanations 
for  teaching  various  types  of  procedural 
tasks,  and  (2)  three  experimental  studies 
of  the  effects  of  presenting  qualitative 
explanations  on  learning  and  retention  of 
an  assembly  procedural  task. 


BACKGROUND 

Most  instruction  for  procedural 
tasks  is  "lean."  That  is,  procedural  tasks 
are  most  often  taught  as  a  linear 
sequence  of  steps  with  a  single  top-level 
goal.  The  instruction  typically  contains 
little  explanation  about  the  system  or 
situation  and  how  the  steps  interrelate, 
and  requires  rote  memorization.  This 
research  investigated  the  effects  of 
presenting  qualitative  explanations  or 
elaborated  instructions  on  learning, 
performance,  and  long-term  retention  of 
procedural  tasks.  Qualitative 
explanations  are  what  Navy  instructors 
and  training  developers  are  talking  about 
when  they  use  the  word  "theory." 
because  of  the  controversy  in 
Instructional  S)  terns  Development  (1SD) 
and  Navy  technical  training  over  how 
much  theory  to  give  students,  it  is 
important  to  determine  when  and  how 
much  theory  should  be  presented  when 
learning  specific  types  of  procedural 
tasks.  The  reason  for  this  is  that  adding 


theory  or  qualitative  explanations  to 
instruction  lengthens  training.  However, 
adding  theory  may  enhance  learning  and 
retention.  The  rationale  for  this  notion  is 
that  qualitative  explanations  enable 
learners  to  build  mental  models  or 
concrete  cognitive  representations  of 
systems  and  tasks. 


Research  has  shown  that  providing 
students  with  supplemental  explanations 
sometimes  facilitates  learning  and 
retention  of  procedural  and  complex  rule- 
and  principle-based  tasks  (Gentner,  1980, 
1981;  Kieras,  1981;  Smith  3c  Goodman, 

1982;  Sturges,  Ellis,  3c  Wulfeck,  1980; 
Tourangeau  3c  Sternberg,  1982).  However, 
in  a  review  of  the  literature,  it  was 
discovered  that  very  little  work  had  been 
done  on  procedural  tasks  and  that  various 
authors  discuss  and  define  qualitative 
explanations,  analogies,  etc.,  and 
procedural  tasks  in  different  ways  (Smith  3c 
Goodman,  1982;  Stevens  3c  Steinberg, 

1981).  Based  on  this  literature  review  an 
operationally  defined  taxonomy  of  types  of 
qualitative  explanations  was  developed. 

The  types  are  linear  (what),  structural 
(how/ why),  and  functional  (how/ why). 
Linear  explanations  provide  the  student 
with  inventory  information  about  what  to 
do  with  the  system  or  in  the  situation, 
structural  explanations  provide  information 
about  how  or  why  the  system  is 
constructed,  and  functional  explanations 
provide  information  about  how  or  why  the 
system/ situation  works.  These  types 
incorporate  the  kinds  of  explanations 
described  by  others  and  they  have  the 
advantage  of  being  relatively  easy  to 
define  operationally. 


After  the  types  of  explanations  were 
identified  and  defined,  the  research  rele¬ 
vant  to  each  type  was  reviewed.  Then  the 
types  of  explanation  necessary  for  learning 
specific  procedural  tasks  were  identified. 
Because  procedural  tasks  differ  in  a 
variety  of  ways,  the  initial  part  of  this 


Table  1 


Type  of  Procedural  Task  and  Type  of  Theory 
That  May  Improve  Retention 


Type  of 

Procedural  Task 

Type  of  Presentation 

Linear 

Structural  Theory 

How  Why 

Functional  Theory 
How  Why 

Operation 

* 

« 

* 

+ 

+ 

Maintenance 

Repair 

» 

+ 

+ 

+ 

+ 

Assembly 

* 

*► 

+ 

+ 

+ 

Paper-based 

Filing  forms 

* 

+ 

Formatting  documents 

* 

+ 

+ 

+ 

Locating  information  or  objects 

* 

+ 

+ 

+ 

4* 

Note.  Qualitative  explanations  (theory)  can  be  direct  or  analogical. 

*  Indicates  that  a  theory  can  be  applied  to  the  procedure. 

"^Indicates  that  teaching  the  theory  behind  the  procedure  may  facilitate  performance/ 
retention. 


process  involved  identifying  the  types  of 
procedural  tasks.  Four  different 
"classes"  of  tasks  were  identified:  (1) 
operator,  (2)  maintenance/ repair/ 
assembly,  (3)  paper-based,  and  (4) 
locating  information  or  locating  objects. 
Table  1  shows  the  types  of  explanations 
that  can  be  used  with  the  various  types 
of  procedural  tasks. 

APPROACH 

One  possible  solution  to  the  problem 
of  information  retention  that  has  not 
received  much  attention  is  the  notion 
that  qualitative  explanations,  elaborated 
instructions  and  analogies  enable 
learners  to  build  mental  models  or 
representations  of  systems  and  tasks, 


which  in  turn  facilitate  learning,  per¬ 
formance,  and  retention.  The  approach  of 
this  research  was  to  examine  empirically 
the  effects  of  various  types  of  explanations 
on  learning  and  retention  of  procedural 
tasks. 


The  task  consisted  of  assembling  a 
model  crane.  The  subjects,  enlisted 
military  personnel  (experiments  I  and  Ill) 
and  college  undergraduates  (experiment  II), 
were  randomly  assigned  to  either  a 
linear/structural  instruction  group,  a 
functional  instruction  group,  or  picture 
only  group,  and  were  scored  on  speed  and 
accuracy  of  constructing  the  model. 
Immediate  and  delayed  (one  month) 
performance  testing  was  administered  to 
all  subjects. 
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RESULTS  AND  DISCUSSION 

The  results  from  experiments  I  and 
II  indicated  that  the  subjects  who 
received  functional  qualitative 
instructions  were  better  at  recall  and 
performance  of  the  model  assembly  than 
the  subjects  who  received  linear/ 
structural  qualitative  explanations. 
However,  results  from  experiment  III  did 
not  show  such  an  advantage  for  the 
functional  qualitative  explanations  over 
the  linear/structural  explanations.  The 
findings  in  experiment  III  are  probably 
the  result  of  a  ceiling  effect.  The 
military  subjects  in  experiment  III 
reported  on  an  experience  questionnaire 
that  they  had  more  mechanical 
experience  than  did  the  college  subjects 
in  experiment  II  (experiment  I  subjects 
did  not  receive  an  experience 
questionnaire).  The  less  experienced 
college  subjects  (experiment  III)  and 
military  subjects  from  a  nontechnical 
ratings  (experiment  I)  appeared  to 
benefit  more  from  the  qualitative 
functional  instructions  than  did  the 
experienced  military  subjects 
(experiment  II),  perhaps  because  the 
military  subjects  in  experiment  III 
already  had  enough  mechanical 
experience  to  make  the  functional 
explanations  redundant. 


CONCLUSIONS 

In  summary,  providing  additional 
explanations  can  be  beneficial. 

However,  the  different  results  for 
different  levels  of  experience  point  out 
the  importance  to  training  designers  in 
considering  prior  experience  and 
background  variables  when  prescribing 
guidelines  for  instruction.  Instructional 
designers  are  encouraged  to  consider  the 
interactions  between  the  learner's 
knowledge  structures,  the  structure  of 
the  information  content,  and  the 
structure  of  the  task  when  developing 
instruction. 


Instructional  developers  should  consider 
the  "trade-offs"  in  training  time  and 
performance  for  the  various  types  of 
explanations  when  designing  instruction. 
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REWARD  SYSTEM  DESIGN  AND  PERFORMANCE  ' 

Delbert  M.  Nebeker 


Productivity  improvement  has  become  vital  to  the  Navy’s  success  in 
meeting  its  defense  commitments  because  of  the  expansion  of  fleet  operations 
with  tighter  fiscal  and  manpower  constraints.  Navy  managers  have, 
therefore,  made  productivity  improvement  of  organizations  and  employees  a 
high  priority.  Some  innovative  reward  systems  to  motivate  and  guide 
productivity  have  been  successful.  However,  there  is  controversy  over  whether 
or  not  performance  standards  should  be  easy  or  difficult,  and  the  size  and 
types  of  rewards.  For  these  questions,  no  acceptable  evidence  suggests  which, 
if  any,  of  the  many  alternatives  is  best  and  when  it  should  be  used.  This 
research  compared  the  effectiveness  of  several  different  reward  systems.  The 
combination  of  easy  performance  standards  and  larger  rewards  for  exceeding 
these  standards  produced  the  largest  performance  improvements.  A  second 
experiment  is  underway  and  will  be  completed  in  early  FY86.  All  of  the 
systems  produced  cost  savings  as  compared  to  straight  wage  payment,  and 
several  produced  larger  performance  improvements  than  the  others. 


PROBLEM 

Improving  the  productivity  of  Navy 
organizations  and  employees  is  a  high 
priority  for  Navy  managers.  Increasingly 
severe  fiscal  and  manpower  constraints 
during  the  expansion  of  fleet  operations 
make  productivity  improvement  vital  to 
the  Navy's  success  in  meeting  its  defense 
commitments. 


BACKGROUND 

Several  programs  have  successfully 
increased  productivity  in  Navy  organi¬ 
zations  by  using  innovative  reward 
systems  to  motivate  and  guide 
productivity  improvements  (Nebeker  <3c 
Neuberger,  1985;  Shumate,  Dockstader, 
6c  Nebeker,  1978).  However,  despite 
these  successes,  we  know  very  little 


*CoM  shared  with  Center  (,.2  funding  ($50K  from  PE  6276JN), 


about  designing  optimal  reward  systems. 
The  recommendations  found  in  research 
and  in  practice  on  the  proper  design  of 
these  systems  are  contradictory.  For 
example,  serious  conflict  exists  over 
whether  easy  or  difficult  performance 
standards  should  be  used  to  determine  who 
is  rewarded.  Likewise,  disagreement  and 
controversy  abound  over  the  size  and  type 
of  the  rewards.  Other  questions  are  raised 
about  how  to  tie  the  amount  of  the  reward 
to  the  amount  of  the  improvement  in  per¬ 
formance.  For  these  and  other  questions 
about  the  parameters  of  reward  systems, 
there  is  no  acceptable  evidence  suggesting 
which,  if  any,  of  the  many  alternatives  is 
best  and  when  it  should  be  used. 

This  research  compares  the 
effectiveness  of  several  performance 
standards  and  rewards  on  employee 
productivity. 
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APPROACH 

This  research  was  conducted  in  the 
Organizational  Systems  Simulation 
Laboratory  (OSSLAB)  at  the  Navy 
Personnel  Research  and  Development 
Center.  The  OSSLAB  simulates  working 
conditions  and  hires  actual  employees, 
who  work  under  a  variety  of 
management  practices  and/or  styles. 

The  experimental  and  laboratory 
conditions  used  make  it  possible  to 
collect  extensive  and  accurate 
performance  and  behavioral  data  on  the 
workers.  Actual  observed  differences  in 
productivity,  as  well  as  activity, 
satisfaction  and  stress,  provide  the  bases 
for  developing  theory  and  recommending 
field  applications. 


This  experiment  investigated  the 
joint  effects  of  the  difficulty  of 
performance  standards  and  the 
magnitude  of  rewards.  Twenty-four 
proficient  keyboard  operators  were 
recruited  and  hired  (at  $4.89  per  hour)  to 
enter  and  maintain  references  in  a  data 
base  for  searching  and  retrieving  the 
scientific  literature.  The  employees 
were  divided  into  two  equal  shifts  that 
worked  two  4-hour  shifts  a  week  for 
eight  weeks.  A  work-sample  test  given 
to  measure  ability  revealed  substantial 
variability  within  shifts,  but  no 
significant  differences  between  the  two 
shifts. 


Both  shifts  performed  in  a  baseline 
or  control  condition  before  the 
introduction  of  performance  standards 
and  financial  incentives.  Once 
performance  had  stabilized  in  the 
baseline  condition,  performance 
standards  were  set  for  both  shifts.  One 
shift  was  assigned  a  difficult 
performance  standard,  which  was  set  at 


the  90th  percentile  of  the  shift's  baseline 
performance  (6600  keystrokes  per  hour). 
The  second  shift  was  assigned  an  easy 
standard,  which  was  set  at  the  20th 
percentile  of  their  baseline  performance 
(4800  keystrokes  per  hour).  When 
employees  exceeded  these  performance 
standards,  they  qualified  for  a  reward.  The 
magnitude  of  the  reward  depended  upon 
the  experimental  condition.  Both  shifts 
were  offered  small  incentives  during  weeks 
3  through  5  and  large  incentives  during  the 
last  three  weeks.  The  small  incentive  was 
a  15-percent  "sharing  rate";  that  is,  15 
percent  of  the  labor  cost  savings  from 
performance  above  standard  were  "shared" 
with  the  responsible  employee.  For 
example,  an  employee  performing  10 
percent  above  the  standard  received  a  1.5 
percent  increase  in  hourly  pay.  The  large 
incentive  was  a  50-percent  sharing  rate; 
an  employee  performing  10  percent  above 
the  standard  earned  a  5  percent  increase  in 
pay. 


RESULTS  AND  CONCLUSIONS 

The  results  proved  to  be  very 
interesting  in  view  of  the  fact  that  many 
researchers  have  argued  for  difficult 
performance  standards  (e.g.,  Locke,  Shaw, 
Saari,  &  Latham,  1981).  We  found  that  the 
combination  of  easy  performance  standards 
(80%  of  employees  were  already  exceeding 
the  standard)  and  larger  incentives  for 
exceeding  these  standards  (50%  sharing 
rate)  produced  the  largest  performance 
improvements  (see  Figure  1).  Performance 
under  difficult  standards  improved  over  the 
baseline  but  was  not  affected  by  the 
magnitude  of  the  incentive  offered.  It  is 
interesting  to  note  that  all  the 
combinations  of  reward  amounts  and 
performance  standards  produced 
substantial  cost  savings  (from  14%  to  24%) 
over  straight  wages  paid  in  the  baseline 
condition. 
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Figure  1.  Employee  performance  a?  a  function 
of  incentive*  offered  and  difficulty  of 
performance  standard 


From  these  data,  as  well  as  other 
evidence,  wc  conclude  that: 

1.  Setting  difficult  performance 
standards  is  not  justified  by  performance 
improvement  or  cost  considerations. 
Assuming  employee  acceptance  of  the 
performance  measurement  system  is 
desired,  the  use  of  difficult  standards  is 
even  less  justified. 

2.  Under  linear  sharing  rate 
systems,  such  as  the  one  used  here,  the 
optimum  magnitude  of  the  rewards  is 
approximately  50  percent  of  the  wages 
saved  by  exceeding  productivity 
standards. 


3.  Any  of  the  designs  used  here 
promise  large  cost  savings  when  properly 
implemented.  Perhaps  it  is  this 
generalized  success  across  a  variety  of 
designs  that  has  produced  contradictory 
recommendations  in  the  literature  to  date. 
W'thout  a  direct  comparison  between 
competing  alternatives,  such  as  was  done 
here,  it  is  difficult,  if  not  impossible,  to 
determine  which  particular  design  is  most 
effective. 

PLANS 

A  second  experiment  in  the  OSSLAB 
begun  in  FY85  will  be  completed  in  early 
FY86  under  exploratory  development 
funding.  This  experiment  compares  the 
effectiveness  of  increasing  the  magnitude 
of  reward  in  accelerating  amounts  instead 
of  constant  amounts  for  incremental 
increases  in  performance.  Evidence 
supporting  the  superiority  of  the 
accelerating  magnitudes  of  rewards  would 
have  a  significant  impact  upon  both 
practical  and  theoretical  thinking. 
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RELATIONSHIPS  BETWEEN  MANAGEMENT  PRACTICES 
AND  ORGANIZATIONAL  PERFORMANCE 


Mark  C.  Butler 
Kent  S.  Crawford 


The  internal  management  practices  of  two  Navy  logistics  organizations 
similar  in  size,  structure,  personnel  composition,  and  function,  but  differing 
in  effectiveness  were  examined.  Managers  in  the  more  effective  organization 
maintained  efficiency  and  adapted  to  change  by  using  creative  and  flexible 
management  practices  to  meet  their  facility’s  goals. 


PROBLEM  productivity  and  performance.  Specif¬ 

ically,  they  want  to  know  why  some 

Navy  logistics  organizations  are  organizations,  or  organizational  subunits, 

faced  with  the  challenge  of  meeting  perform  more  effectively  than  others  even 

increasing  demands  with  decreasing  though  all  have  similar  structure,  size, 

resources.  At  the  same  time,  major  personnel  composition,  and  function.  To 

developments  in  technology,  manage-  address  this  question,  some  researchers  are 

merit  innovations  in  business  and  examining  the  impact  of  external  factors — 

industry,  and  increasing  administrative  those  outside  the  immediate  organizational 

constraints  including  increased  context-on  the  effectiveness  of  an  organi- 

accountability  are  also  changing  the  zation.  For  example,  consumer  assessment 

logistics  environment  of  the  1980s.  of  organization  performance  is  currently 

viewed  as  an  important  and  frequently 
overlooked  externally  derived  effec- 

Implementation  of  organizational  tiveness  factor  CJobson  <5c  Schneck,  1982; 

improvements  and  innovative  manage-  Zamrnuto,  1984).  Other  researchers  argue 

ment  practices  could  improve  operation  persuasively  that  the  traditional  emphasis 

of  Navy  logistics  organizations.  on  studying  the  impact  of  internal  factors, 

However,  we  do  not  clearly  understand  such  as  organizational  goals  and  problem 

those  organizational  and  managerial  solving  style,  should  be  retained  and 

factors  that  contribute  to  effec-  expanded  to  include  assessing  organization 

tiveness  under  conditions  of  accelerated  culture  (Kiefer  <5c  Stroh,  1983). 
change.  The  practices  with  the  greatest 
potential  for  improving  mission 

accomplishment  need  to  be  identified  Although  the  Navy's  logistic  organiza- 

and  evaluated.  tions  operate  within  a  rapidly  changing 

environment,  their  structure,  technology, 
personnel  composition,  and  reporting 
requirements  (i.e.,  chain  of  command)  are 

BACKGROUND  similar.  To  understand  differences  in 

performance  and  effectiveness,  we  need  to 

Increasingly,  organizational  determine  areas  in  which  homogeneous 

researchers  and  practicing  managers  are  logistics  oiganizations  have  actual  control 

concerned  with  issues  related  to  or  discretion.  In  this  instance,  internal 
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Effective  management  of  people  makes  a  significant  contribution  to  productivity  In 
Navy  industrial  organizations. 


issues  related  to  organizational 
effectiveness  rather  than  external 
management  factors  should  be  examined 
because  external  factors  are  heavily 
constrained  by  higher  level  policies. 

Determining  the  relationship  of  a 
number  of  management  practices  to 
organizational  performance  would 
facilitate  development  of  a  model  of 
organizational  functioning  to  identify 
the  management  practices  that  enhance 
individual,  subunit,  and  organizational 
performance.  Such  a  model  would 
recognize  that  some  management 
practices  are  specific  to  certain 
settings,  while  others  are  more  generic 
and  applicable  across  settings.  In  short, 
some  management  practices  probably  (1) 
vary  across  organizational  settings,  (2) 
are  unique  to  certain  settings,  and  (3) 
are  associated  more  with  high  versus  low 
performance  effectiveness. 


GOAL 


The  specific  goal  of  the  present  study 
was  to  identify  the  internal  management 
practices  that  differentiate  between 
effective  and  ineffective  logistics 
organizations. 


APPROACH 


To  identify  key  management  prac¬ 
tices,  researchers  selected  two  large  naval 
industrial  organizations  for  study— one 
traditionally  performing  at  a  high  level  of 
effectiveness;  and  the  other,  at  a  low 
level.  Selection  was  based  on  the  same 
institutional  criteria  that  the  naval 
headquarters  command  uses  to  evaluate 
the  organizations'  ability  to  meet  their 
stated  goals. 
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Interviews  with  key  managers  at 
both  organizations  identified  the 
significant  management  practices 
currently  in  use.  The  interview  data 
were  also  the  basis  for  development  of 
an  extensive  management  practices 
questionnaire  (MPQ),  which  was 
administered  to  comparable  groups  of 
managers  in  each  setting. 


RESULTS 

Two  major  areas  distinguish  the 
participating  organizations  from  each 
other— goals  and  management  practices. 
In  the  area  of  goals,  top  management  in 
both  facilities  reported  that  the  success 
of  their  respective  organizations 
depended  on  maintaining  efficiency  and 
adapting  to  change.  While  both 
participating  facilities  emphasized 
achieving  these  goals  by  adhering 
carefully  to  "bottom-line"  outcomes  such 
as  schedule,  cost  and  quality  in  the  more 
effective  facility,  there  was  an 
increased  emphasis  on  improving  internal 
efficiency  (e.g.,  exceeding  standards, 
reporting  accurate  information), 
eliminating  productivity  impediments, 
and  enabling  personnel  development.  In 
the  area  of  management  practices, 
managers  in  the  more  effective 
organization  reported  engaging  in 
proactive,  ongoing  development 
activities,  having  a  clear  understanding 
of  the  structure  and  purpose  of  their 
organization,  stressing  a  relatively  high 
degree  of  coordination  of  work  flow,  and 
possessing  adequate  authority  and 
information  to  perform  their  respective 
managerial  functions. 


CONCLUSIONS 

Managers  in  the  more  effective 
organizations  differed  from  managers  in 
the  less  effective  organizations  in  the 
goals  they  emphasized  as  well  as  the 
management  practices  they  employed  to 
achieve  those  goals.  The  effective 
organizations  seem  to  encourage  and 
support  greater  creativity  and  flexibility  in 
managerial  decision  making.  Future 
efforts  will  attempt  to  replicate  these 
findings  in  less  homogeneous  organizations, 
as  well  as  examine  how  individual 
difference  measures  moderate  these 
findings. 
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ANALYSI5  OF  COGNITION  IN  NATURAL  SETTINGS 

Edwin  Hutchins 


Most  thinking  jobs  in  the  Navy  are  conducted  by  groups  of  people  working 
together,  but  the  study  of  cognition  has  traditionally  used  the  individual  in 
isolation  as  the  unit  of  analysis.  Very  little  is  known  of  the  properties  of 
systems  of  socially  distributed  cognition  and  appropriate  theory  and  methods 
for  studying  such  systems  are  currently  lacking.  This  research  took  as  its 
basic  unit  for  analysis  the  piloting  team  of  a  large  Navy  ship.  This  approach 
enabled  much  of  the  internal  behavior  of  the  system  to  be  observed  in  a  way 
that  is  not  possible  with  processes  internal  to  individuals.  The  results 
revealed  some  fundamental  differences  between  individual  and  socially 
distributed  cognition.  An  understanding  of  the  nature  of  cognition  in  real 
world  settings  is  important  for  diagnosing  performance  inadequacies, 
training,  designing  job  performance  aids,  and  human  factors  design  of 
operational  environments. 


BACKGROUND 

Most  thinking  jobs  in  the  Navy  as 
well  as  in  nonmilitary  settings  are 
accomplished  by  groups  of  people 
working  together.  The  research 
literature  in  cognitive  anthropology  and 
cognitive  psychology  that  concerns 
naturally  situated  cognition  assumes  as  a 
primary  unit  of  analysis,  the  individual  in 
isolation  from  other  individuals  as  well 
as  from  a  world  of  action  and  mediating 
technologies.  In  the  most  interesting 
and  characteristically  human  activities, 
however,  the  cognitive  system  clearly 
includes  two  or  more  minds,  a  shared 
culturally  meaningful  task,  a  social 
organization,  and  technologies  such  as 
literacy  that  mediate  and  externalize 
the  performance  of  individual  cognition. 
The  cognitive  properties  of  such  systems 
of  socially  distributed  cognition  are 
certain  to  differ  in  impor  tant  ways  from 
the  properties  of  the  cognition  of 
individuals.  Yet,  we  know  very  little 
about  how  such  systems  actually  work. 


PROBLEM 

An  understanding  of  the  nature  of 
cognition  in  real  world  settings  is 
important  for  (1)  diagnosing  performance 
inadequacies,  (2)  providing  training  that  is 
relevant  to  the  actual  job  to  be  performed 
(3)  designing  performance  aids,  and  (4) 
designing  operational  work  environments. 
Furthermore,  as  work  environments 
become  more  automated  and  include  more 
intelligent  systems  that  interact  in 
fundamentally  cognitive  ways,  we  must 
develop  an  understanding  of  how  systems 
of  distributed  cognition  work.  At  present, 
however,  a  lack  of  appropriate  theory  and 
method  hampers  our  understanding  of 
cognition  in  settings  outside  the 
laboratory. 


APPROACH 

The  first  step  in  the  project  was  to 
select  a  task  domain  in  which  a  system  oi 
socially  distributed  cognition  is  in 
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operation.  The  piloting  team  on  the 
bridge  of  a  large  Navy  ship  was  selected 
because  the  following  important  proper¬ 
ties  make  it  amenable  to  study: 

1.  A  group  of  six  individuals 
jointly  performs  a  few  culturally  well- 
defined  behaviors.  Therefore,  we  know 
that  the  task  is  socially  distributed  and 
what  is  supposed  to  be  done  is  known 
precisely. 

2.  Several  mediating  technologies 
and  representations  are  utilized.  These 
include  literacy,  the  use  of  charts, 
graphs,  and  other  external  computational 
media  as  well  as  electronic  devices  that 
perform  parts  of  the  computation. 

3.  The  principal  investigator  has 
more  than  25  years  of  experience  with 
the  navigation  procedures  involved.  This 
means  that  the  researcher  could  easily 
achieve  a  degree  of  credibility  in  the 
eyes  of  the  team  under  investigation  and 
that,  during  the  analysis  process,  the 
researcher  could  concentrate  on 
discovering  the  properties  of  the  socially 
distributed  cognitive  system  rather  than 
on  trying  to  figure  out  what  the  team 
was  trying  to  do. 


The  most  fundamental  theoretical 
innovation  of  the  approach  taken  here  is  in 
the  boundaries  of  the  cognitive  system  that 
is  to  be  the  unit  of  analysis.  Rather  than 
looking  primarily  at  a  single  individual  and 
then  trying  to  see  how  a  team  of 
individuals  could  function  jointly,  this 
project  took  as  the  fundamental  unit  a 
group  of  people  jointly  performing  a  single 
task.  Communication  among  the  group 
members  is  seen  as  an  internal  process  of 
the  cognitive  system;  computational  media 
(e.g.,  charts),  as  internal  representations  of 
the  system;  and  the  computations  carried 
out,  more  as  internal  processes  of  the 
system.  Yet,  because  the  cognitive 
activity  is  distributed  across  a  social 
network,  these  internal  processes  and 
internal  communications  are  directly 
observable.  If  a  cognitive  psychologist 
could  get  inside  a  human  mind,  he  or  she 
would  want  to  look  at  the  nature  of  the 
representation  of  knowledge,  the  nature 
and  kind  of  communication  among 
processes,  and  the  organization  of  the 
information  processing  apparatus.  In  such 
a  fantasy,  the  underlying  processes  would 
be  obscured  at  some  level  of  detail.  If 
these  observations  could  be  made, 
however,  any  cognitive  psychologist  would 
be  very  happy.  With  systems  of  socially 


Mott  thinking  jobs  in  the  Navy  are  conducted  by 
groups  ol  people  *vort<lng  together. 


27 


aTT  ^ .^LVATA5«k.,I Jt-I  J5«  X  «DU4  «rw  ~*  k  *  jr-. 


distributed  cognition  we  CAN  step  inside 
the  cognitive  system.  The  analysis  in 
this  project  begins  by  attending  precisely 
to  things  that  the  cognitive  psychologist 
would  attend  to  if  he  or  she  were  given 
the  opportunity  to  step  inside  a  mind. 

The  data  consist  of  written  and 
audio-and  videotaped  observations  of  the 
behavior  of  the  piloting  team  on  the 
bridge  of  a  Navy  ship  in  high  workload 
situations.  The  system's  response  to  a 
true  crisis  situation  was  also  observed 
and  recorded.  Data  were  collected 
during  two  at-sea  periods  in  February 
and  April  1985.  Transcription  and 
analysis  of  the  data  are  now  in  progress. 


DISCUSSION  AND  RESULTS 

Initial  analyses  of  the  data  show 
that  detailed  accounts  of  the  internal 
states  of  the  cognitive  system  can  be 
constructed.  Some  aspects  of  the 
behavior  of  the  distributed  system 
mirror  what  we  already  know  about 
individual  cognition.  The  approach  used 
here  makes  a  good  deal  of  the  internal 
behavior  of  the  system  observable  in  a 
way  that  individual  psychological 
processes  are  not.  Much  of  what  we  see 
resembles  what  has  been  inferred  about 
individual  cognitive  processes.  There¬ 
fore,  this  research  promises  to  provide  a 
wealth  of  interesting  hypotheses  about 
the  nature  of  individual  cognition. 

Still,  the  properties  of  socially 
distributed  cognition  differ  from  the 
properties  of  individual  cognition  in 
important  ways.  For  example,  suppose  a 
procedure  requires  actions  at  several 
different  locations.  While  an  individual 
task  performer  must  distribute  those 
activities  across  time,  a  socially 
distributed  cognition  system  may 
distribute  activities  simultaneously 
across  space.  Socially  distributed 
systems  also  permit  the  simultaneous 
maintenance  of  different  interpretations 


of,  or  competing  hypotheses  about,  the 
current  situation.  The  nature  of  the 
discovery  and  resolution  of  such  conflicts 
through  social  interaction  contrasts  with 
the  search  for  confirming  information 
generally  observed  in  individual  cognition. 

Finally,  when  a  procedure  is 
distributed  across  a  group  of  individuals, 
procedural  dependencies  become  social 
dependencies.  While  communication  within 
a  single  mind  seems  to  be  relatively  rapid, 
communication  between  individuals  always 
depends  upon  the  information-carrying 
properties  of  some  external  medium  (i.e., 
spoken  language  or  a  diagram).  This  has 
implications  for  how  to  distribute  tasks 
usefully  across  individuals.  Consider  the 
task  of  figuring  out  which  features  of  the 
world  correspond  to  which  features  on  a 
chart.  This  task  is  not  always  easy  for  one 
person  to  do  even  when  that  person  can 
consult  both  the  chart  and  a  representation 
of  the  world.  However,  the  task  becomes 
almost  impossible  when  it  is  distributed 
across  two  individuals  so  that  one  has 
access  to  the  chart,  the  other  has  access  to 
the  world,  and  they  have  access  to  each 
other  only  via  spoken  language  (this 
situation  sometimes  arises  aboard  ship). 
This  difficulty  is  caused  by  the  bandwidth 
of  communication  required  between  the 
representation  of  the  chart  and  the 
representation  of  the  world  being  greater 
than  that  provided  by  the  medium  of 
communication  between  the  two  task 
performers. 


PLANS 

The  transcription  and  analysis  of  the 
data  will  continue  in  FY86.  As  some 
aspects  of  the  behavior  of  these  systems 
seem  amenable  to  simulation  modeling,  an 
attempt  will  be  made  to  construct  such 
models.  The  theory,  method,  and  example 
analyses  will  be  written  up  as  a  series  of 
technical  reports  and  as  a  monograph  for 
publication.  No  further  field  research  is 
planned  at  this  time. 


EXPERT  SYSTEMS  FOR  FAULT  DIAGNOSIS* 
STOCHASTIC  PROCESSES 


Donald  B.  Malkoff 


Lack  of  precise  information  needed  for  reliable  fault  diagnosis  and  lack 
of  time  in  which  to  make  the  diagnosis  and  act  on  it  make  coping  with 
malfunctions  during  the  operation  of  complicated  control  systems  difficult  for 
Navy  personnel.  The  automated  support  operators  need  for  fault  diagnosis 
and  decision  making  generally  involves  the  use  of  computer  "< expert  systems." 

A  method  is  proposed  for  the  development  of  computer  programs  that 
automatically  refine,  correct,  and  complete  the  information  used  for  decision 
making  and  automatically  associate  the  pattern  of  system  behavior  following 
a  malfunction  with  its  eventual  diagnosis.  New  computer  hardware  is  on 
order  to  simulate  malfunctioning  of  a  real-time  control  system  using  the 
DD  963  ship  propulsion  lubrication  oil  system  as  a  test  bed. 


BACKGROUND  The  application  of  expert  systems  to 

the  handling  of  malfunctions  in  real-time 

Navy  personnel  encounter  great  control  systems  has  been  problematic.  The 

difficulties  in  coping  with  malfunctions  information  needed  for  diagnosis  by  the 

during  the  operation  of  complicated  computer  program  is  often  either  missing, 

real-time  control  systems.  In  such  imprecise,  or  incorrect.  Additionally,  the 

systems,  the  time  required  for  a  environmental  state  of  the  process  to  be 

response  may  be  critical;  the  behavior  of  controlled  may  be  changing  unpredictably 

the  system  is  influenced  by  random  from  time  to  time,  causing  significant 

factors  in  the  environment;  and  variability  in  its  behavior  in  response  to 

information  overload  exists.  As  a  result,  malfunctions, 
the  total  knowledge  necessary  for  either 

a  human  operator  or  a  computer  expert  A  possible  solution  to  the  problem  of 

system  to  make  reliable  diagnoses  is  insufficient  information  for  the  diagnosis 

often  not  available.  of  malfunctions  lies  in  the  development  of 

computer  programs  that  are  able  to  (1) 

Attempts  to  provide  the  operator  automatically  refine,  correct,  and 

with  automated  support  have  generally  complete  the  information  data  base  used 

dealt  with  well-defined  routine  for  decision  making,  and  (2)  automatically 

procedures  for  system  start-up  and  associate  the  pattern  of  system  behavior 

shutdown.  The  most  pressing  need,  following  a  malfunction  with  its  eventual 

however,  is  for  automated  support  for  diagnosis.  Computer  programs  having  this 

the  tasks  of  real-time  fault  diagnosis  and  capability  are  said  to  be  able  to  perform 

decision  making.  Automated  diagnosis  "pattern  recognition."  In  the  case  of  gas 

and  decision  making  ordinarily  involve  turbine  engine  control,  the  elements  that 

the  use  of  computer  "expert  systems."  make  up  the  patterns  would  include 
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sequences  of  the  specific  values  of 
system  variables  such  as  temperature, 
flow  rates,  rotational  speed,  etc.,  of 
various  engine  components.  Programs 
using  pattern  recognition  can 
theoretically  learn  by  association  which 
patterns  are  indicative  of  specific 
malfunctions  and  continuously  update 
these  associations  so  as  to  adapt  to 
changing  conditions. 


PROBLEM 

Pattern  recognition  is  not  presently 
possible  in  systems  where  the  timing  of 
events  following  a  malfunction  is 
affected  in  unpredictable  fashions  by  a 
multitude  of  unknown  factors.  For 
example,  on  one  occasion  that  an  oil 
reservoir  begins  to  leak  rapidly,  the  first 
sign  of  trouble  may  be  the  triggering  of 
a  low  oil  fluid-level  alarm.  On  the  next 
instance  of  this  malfunction,  the  leak 
may  be  slower,  in  which  case  the  first 
sign  of  trouble  may  be  a  low  oil  pressure 
alarm,  or  perhaps  a  high  temperature 
alarm  in  the  component  being  supplied 
by  that  oil  reservoir.  If,  on  every 
occasion  of  the  same  type  of 
malfunction,  its  pattern  of 
symptomatology  differs,  how  can  a 
computer  expert  system  possibly  learn  to 
diagnose  the  cause?  This  dilemma  is 
referred  to  as  the  "stochastic  problem." 


It  will  not  suffice  for  the  computer 
program  to  deal  with  the  stochastic 
problem  by  attempting  to  remember  all 
of  the  previous  patterns  that  followed  all 
previous  malfunctions.  Since  there  is  an 
infinite  variety  of  patterns  that  can 
occur,  the  computer's  memory  will  soon 
be  filled  and  unable  to  function. 
Furthermore,  each  new  occurrence  of  a 
malfunction,  because  of  the  random 
factors,  will  be  somewhat  different  than 
any  of  the  previous  patterns  stored  in 


memory.  An  exact  match  will  never  be 
possible.  The  computer  expert  system  may 
not  be  able  to  recognize  it  as  a  familiar 
pattern. 


APPROACH 

In  the  method  we  propose  for  dealing 
with  the  stochastic  problem,  the  pattern  of 
system  variable  values  following  a 
malfunction  is  viewed  as  a  sequence  of 
significant  events,  for  example,  of  the 
triggered  alarms.  The  sequence  is 
characterized  in  three  ways:  (1)  the 
specific  alarms  that  are  triggered  by  a 
malfunction,  (2)  the  order  in  which  those 
alarms  are  triggered,  and  (3)  the  amount  of 
time  elapsed  between  successive  alarms. 

When  a  new  occurrence  of  an  as-yet 
undiagnosed  malfunction  produces  a 
developing  sequence  of  alarms,  the 
sequence  is  NOT  directly  compared  with 
previous  sequences  in  memory.  Instead,  a 
list  is  constructed  of  all  malfunctions  that 
have  been  known  in  the  past  to  cause  this 
particular  order  of  these  particular  alarms. 
At  this  stage,  no  attention  is  paid  to 
elapsed  times  between  alarms. 


For  each  of  the  malfunctions  on  the 
list,  a  binomial  probability  curve  is 
constructed  based  on  that  malfunction's 
recent  history  of  time  intervals  for  each 
alarm  in  the  sequence.  In  other  words,  all 
previous  manifestations  of  the  same  kind 
of  malfunction  are  averaged  and 
incorporated  into  a  single  curve  for  every 
event  or  alarm  in  the  sequence.  Now  the 
newly  occurring,  undiagnosed  sequence  can 
easily  be  compared  with  each  of  the  curves 
to  determine  the  best  match.  In  this 
manner,  even  before  the  sequence  is 
complete,  the  most  probable  diagnosis  can 
be  made  at  every  step  of  the  way  with 
increasing  accuracy  as  time  elapses 
following  the  onset  of  the  malfunction. 
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While  the  description  of  this  method 
may  sound  complex  and  be  difficult  to 
envision,  it  is  remarkably  similar  to  the 
methodology  of  medical  diagnosis.  The 
physician,  in  attempting  to  make  a 
clinical  diagnosis  based  on  the  patient's 
symptoms  and  on  the  clinical  syndrome 
(the  order,  timing,  and  severity  of  the 
symptoms),  compares  the  symptom¬ 
atology  of  the  present  case  with  his 
model  or  conception  of  the  TYPICAL,  or 
USUAL  syii.ptomaiology  of  various 
diseases.  He  recognizes  that  the 
specific  symptomatology  of  a  patient 
will  seldom  exactly  match  with  any 
previous  case.  His  models  are  nothing 
more  than  mental  (or  textbook) 
reconstructions  of  averages  for  the  most 
probable  sequences  of  symptoms,  much 
in  the  manner  of  the  sequence  of 
probability  curves  for  gas  turbine  engine 
alarms  in  our  approach. 


RESULTS 

Preliminary  paper-and-pencil 
analysis  indicates  this  approach  to  be  an 
effective  one. 


PLANS 


New  computer  hardware  has  been 
requisitioned  that  will  allow  the  simulation 
of  a  malfunctioning  real-time  control 
system.  In  particular,  it  will  allow  the 
incorporation  of  random  variability  into 
the  timing  of  the  system  processes.  The 
target  system  will  be  the  lubrication  oil 
supply  to  the  reduction  gears  of  the  gas 
turbine  engine  unit  of  the  DD  963  class  of 
ships.  A  read-time  computer  expert  system 
will  be  developed  and  tested. 


BENEFITS 


Should  the  approach  be  successful,  it 
will  provide  a  possible  solution  to  the 
stochastic  problem  encountered  in 
attempts  to  develop  real-time  expert 
systems,  and  provide  a  mechanism  for 
constructing  expert  systems  that  can  learn 
from  experience  and  adapt  to  changing 
conditions.  Both  of  these  imply 
considerable  potential  benefits  for  military 
and  civilian  control  systems. 
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MULTIPLE  CRITERION  OPTIMIZATION  TECHNIQUES 

Timothy  Liang 


Assigning  over  500,000  active  duty  Navy  enlisted  personnel  in  some  300 
distinct  occupational  specialties  at  9  skill  levels  to  jobs  at  naval  activities  and 
units  all  over  the  world  is  baeed  on  numerous  complicated  factors.  These 
assignment  problems  hold  unique  challenges  for  the  integration  of  multiple 
criteria  decision  making  and  large-scale  optimization  techniques. 

In  FY83/84.  the  Navy  Personnel  Research  and  Development  Center 
developed  a  computationally  efficient  pure  network  flow  algorithm  modified 
to  handle  multiple  objectives,  which  the  Enlisted  Personnel  Management 
Center,  New  Orleans  adopted  as  its  main  allocation! assignment  planning 
tool  for  general  detail  personnel.  However,  other  methodologies  are  needed  to 
solve  the  large-scale  multiple  criterion  Navy  assignment  problem  with 
additional,  more  complex  constraints  such  as  required  training  or  multiple 
skills.  Large-scale  integer  programming  and  embedded  networks  were 
found  to  be  insufficient  for  the  problem.  Work  planned  for  FY86  will  use 
aggregation-disaggregation  theory  to  develop  advanced  techniques  in  more 
global  resource  allocation  and  management. 


PROBLEM 

Navy  personnel  assignment  is  a 
large  complex  and  expensive  operation. 
The  annual  permanent  change-of-station 
(PCS)  budget  for  assigning  over  500,000 
active  duty  enlisted  personnel  to  5,000 
different  naval  activities  and  units  as 
distant  as  Antarctica  and  the  Indian 
Ocean  is  over  $525  million.  Personnel 
are  assigned  to  either  sea  or  shore  duty 
jobs  in  about  30C  distinct  occupational 
specialties  and  at  9  skill  levels. 
"Wholesale"  allocation  of  personnel  to 
fill  quotas  to  major  groupings  of  Navy 
units  distinguished  by  geographical 
regions,  types  of  duty,  etc.  precedes  the 
assignment  or  "detail"  of  individuals  to 
jobs  within  those  unit  groups. 

Detailers  match  people  to  jobs  using 
complex  person/job  eligibility  criteria 
and  numerous  rules,  regulations,  and 
policies  (multiple  criteria).  Detailers 
must  be  concerned  with  the  global 


needs  of  the  Navy  (e.g.,  achieving  an 
aggregate  balance  of  scarce,  highly  skilled 
personnel  among  ships  in  the  Atlantic  and 
Pacific  Fleets),  the  requirements  and 
priorities  of  specific  jobs,  and  the  needs 
of  the  individuals  (e.g.,  preferred  location, 
skill  utilization).  Finally,  detailers  must 
make  accurate  and  effective  decisions 
within  tight  moving-cost  constraints. 


Decisions  are  made  at  the  operational 
level  where  aggregate  resource  alloca¬ 
tion  plans  must  be  linked  with  individual 
assignments  as  well  as  operational  policies 
and  constraints.  Navy  assignment 
"planners"  and  "operators"  must  be  able  to 
make  explicit  quantitative  policy  tradeoffs 
to  negotiate  and  agree  on  the  relative 
importance  of  the  policy  criteria  and  the 
flexibility  in  the  execution  of  policy  goals. 
Navy  assignment  problems  hold  unique 
challenges  for  the  integration  of  multiple 
criteria  decision  making  and  large-scale 
optimization  techniques. 
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BACKGROUND 

The  Navy  Personnel  Research  and 
Development  Center  (NAVPFRSRAND- 
CEN)  developed  the  theoretical  and 
empirical  approach  for  an  integrated 
allocation  and  assignment  system  in 
FY83andFY84.  This  system  uses  a 
computationally  efficient,  pure  network 
flow  algorithm  modified  to  handle 
multiple  objectives.  The  initial 
application  allocated  and  assigned 
general  detail  personnel  (i.e.,  those  who 
have  not  yet  obtained  a  specific  occu¬ 
pational  rating).  Close  contact  with  the 
Enlisted  Personnel  Management  Center 
(EPMAC),  New  Orleans  led  to  numerous 
improvements  in  the  model.  After 
testing,  EPMAC  adopted  the  model  as 
their  main  allocation/ assignment 
planning  tool. 

The  pure  network  approach  is  ade¬ 
quate  for  occupations  that  do  not  require 
advanced  training  or  multiple  skills. 
Other  approaches  must  be  investigated 
and  developed  for  complicated  assign¬ 
ment  problems  such  as  those  that  involve 
classroom  availability  at  training 
facilities.  This  work  will  continue  under 
6.3  funding. 


OBJECTIVE 

The  FY85  objective  was  to  investi¬ 
gate  and  develop  methodologies  to  solve 
the  large-scale,  multiple  criterion 
optimization  Navy-assignment  problems 
when  additional,  more  complex 
constraints  are  added. 


APPROACH 

In  FY85,  solutions  to  solve  the 
large-scale,  multiple  criterion 
optimization  Navy  assig'  ment  problem 


were  sought  by  investigating  the  works  of 
Professor  Glen  Graves  at  the  University  of 
California,  Los  Angeles,  the  leading  expert 
in  large-scale  integer  programming,  and 
also  the  works  of  Professors  Jeffrey 
Kennington  at  Southern  Methodist 
University  and  Richard  McBride  at  the 
University  of  Southern  California,  two 
leading  experts  in  the  field  of  embedded 
networks. 

RESULTS 

The  large-scale  integer  programming 
approach  is  simpler  and  more  straight¬ 
forward  than  the  network  approach  for 
mathematical  formulation  of  the  Navy 
assignment  problem.  However,  the  Navy 
assignment  problem  is  much  bigger  than 
the  current  capacity  of  large-scale,  integer 
programming  codes. 

The  embedded  network  approach 
currently  has  the  capacity  to  handle  the 
size  of  the  Navy  assignment  problem  as 
well  as  the  additional  complex  constraints 
that  a  pure  network  code  cannot  handle. 
However,  this  approach  does  not  yield 
integer  solutions,  which  are  essential  to 
the  assignment  problem. 

PLANS 

In  FY86,  additional  research  is  planned 
to  develop  a  network  approach  to  handle 
large-scale  problems  with  complex 
constraints  and  also  produce  integer 
solutions.  Since  adding  complex 
constraints  to  a  network  reduces  its 
computational  efficiency,  investigation  of 
alternative  methods  will  also  be  continued. 

T  wo  other  related  areas  of  research 
that  will  also  be  pursued  are  the  use  of  (1) 
concepts  of  aggregation-disaggregation 
theory  with  cybernetics  for  more  global 
resource  allocation  planning  and  (2) 
modeling  techniques  to  allow  more 
flexibility  in  controlling  policy  tradeoffs. 
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APPLICATIONS 

Past  research  (FY83/84)  was  applied 
to  an  advanced  development  project 
(Personnel  Assignment  System)  to 
develop  an  integrated  allocation  and 
assignment  system  for  general  detail  and 
nominally  trained  enlisted  personnel. 
Current  research  on  the  6.3  project  is 
aimed  at  developing  similar  systems  for 


skilled  enlisted  personnel  who  require 
extensive  technical  training. 

This  technology  will  also  be  used  with 
concepts  in  aggregation-disaggregation 
theory  to  develop  advanced  techniques  in 
global  resource  allocation  and  management. 
Developing  techniques  for  more  effective 
execution  of  the  PCS  budget  is  just  one 
example  of  an  important  application. 


Navy  detailers  must  consider  multiple  criteria  in  matching  people  to  Jobs. 
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Increased  complexity  in  Navy  combat  systems  has  led  to  critical  problems 
of  system  operability,  increased  demands  on  Navy  users,  and  excessive 
training  requirements.  Integrating  the  human  decision  maker  with 
intelligent  support  systems  could  reduce  information  overload  and  improve 
the  effectiveness  of  combat  systems  operations.  This  research  examined  the 
design  of  intelligent  interfaces  for  systems  in  which  the  human  and  computer 
are  viewed  as  partners  working  together.  Relevant  human  factors  issues  were 
identified  and  explored  in  a  simulated  command  and  control  problem  and  a 
prototype  intelligent  interface,  the  intelligent  tactical  assistant,  was 
developed.  The  work  will  progress  to  advanced  development. 


PROBLEM  AND  BACKGROUND 

A  critical  problem  facing  the  Navy 
is  how  to  operate  its  combat  systems 
more  efficiently  to  realize  their  full 
technical  capabilities  and  operational 
effectiveness.  The  major  cause  of 
inefficient  operation  is  the  extreme 
complexity  of  advanced  weapon  and 
sensor  systems  made  necessary  by  the 
increased  threat.  We  attribute  the  fact 
that  advances  in  computer  technology 
have  not  produced  more  operable 
systems  to  an  emphasis  on  form  and 
literal  content  of  isolated  user-computer 
transactions  rather  than  on  their  purpose 
in  the  larger  system.  As  combat  systems 
become  more  complex,  this  approach 
leads  to  excessive  training  requirements 
and  increased  demands  on  Navy  users. 

Emerging  interest  in  intelligent 
interfaces  suggests  that  the  interaction 
of  two  cognitive  systems,  the  user  and 
computer  software,  must  be  considered. 
This  places  a  new  emphasis  on  knowing 
the  user's  goals  and  intentions,  the 
development  of  new  tools  and 


procedures,  and  the  incorporation  of 
contextual  information.  However,  much  of 
the  thinking  has  focused  on  relatively 
passive  environments  such  as  word  proc¬ 
essing  or  data  base  searches.  While  some 
interface  features  that  maximize  "user- 
friendliness"  transfer  readily  to  combat 
systems,  others  may  not  be  appropriate  in 
high  stress  environments.  Indeed, 
altogether  new  requirements  may  become 
evident  in  the  combat  domain.  Thus,  there 
is  a  need  to  define,  investigate,  and 
demonstrate  concepts  of  intelligent 
interfaces  toward  their  adoption  in  combat 
system  design. 


OBJECTIVE 

The  long-term  operational  objective  of 
this  effort  is  to  enhance  combat  system 
performance  by  developing  knowledge- 
based  aids  for  information  integration  and 
manipulation.  The  technical  objective  is  to 
develop  a  laboratory  test  bed  to  investi¬ 
gate  enhanced  interface  concepts  for 
improved  user-computer  collaboration. 
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APPROACH 

The  approach  is  to  identify  and 
investigate  basic  human  factors  issues 
related  to  integrating  human  decision 
makers  with  intelligent  support  systems. 
The  context  for  this  research  was  a 
simulated  command  and  control  problem 
implemented  in  a  laboratory  test  bed. 
Issues  were  identified,  and  related  man- 
machine  interface  concepts  were 
explored  in  the  laboratory  and  applied 
incrementally  to  the  development  of  a 
prototype  intelligent  interface.  This 
exploratory  development  draws  from  the 
fields  of  cognitive  psychology,  decision 
science,  human  factors,  and  artificial 
intelligence. 


RESULTS 


Determinants  of  Interface  Design 

The  following  four  critical  issues 
influence  the  design  of  the  user-computer 
interface.  The  first  two  are  well 
recognized,  but  the  last  two  are  often  not 
addressed. 


1 .  Task  demands.  The  nature  of  the 
task,  especially  the  goals,  priorities, 
constraints,  and  procedures  associated  with 
it  lead  to  critical  design  decisions  for 
allocating  functions  among  personnel, 
hardware,  and  software.  From  this  basic 


starting  point,  the  design  of  the 
interface  often  falters  when  decision 
support  functions  are  designed  as  sets  of 
largely  disconnected  algorithms.  The 
user  is  often  overwhelmed  when  forced 
to  cope  with  their  proper  utilization  and 
management. 

2.  Hardware  and  software  options. 
The  availability  of  hardware  and  soft¬ 
ware  limits  the  ultimate  form  and 
capability  of  the  interface  design.  More 
importantly,  the  selection  of  hardware 
and  software  concepts  determines  the 
type  and  extent  of  intelligent 
interactions  that  the  interface  can 
accommodate. 


3.  Model  of  the  user.  The  system 
designer's  "model"  of  the  user--be  it 
implicit  or  explicit— reflects  the  user's 
abilities,  limitations,  biases, 
preferences,  and  experience, 
inappropriate  or  inaccurate  models  lead 
to  poorly  designed  systems  that 
overestimate,  underestimate,  or  ignore 
human  performance  factors  that  system 
support  functions  could  accommodate. 

4.  Model  of  the  system.  The 
model  of  the  system  that  the  user 
acquires  by  training  and  hands-on 
experience  is  best  thought  of  as  the 
metaphor  about  the  system's  functional 
behavior  that  is  conveyed  to  the  user.  In 
the  worst  case,  the  user  perceives  the 
system  as  a  "black  box."  When  Its 
behavior  is  unsatisfactory  or  unexpected, 
the  user's  first  response  may  be  to  ignore 
it  or  to  turn  it  off.  In  general,  a  well 
chosen  and  well  executed  design 
metaphor  will  reduce  training  require¬ 
ments,  increase  user  satisfaction,  and 
enhance  system  operability. 


Command  and  Control  Environment 

Consider  how  the  user-computer 
interface  issues  apply  to  the  command  and 
control  environment.  What  are  their 
implications  for  the  design  of  interfaces 
for  combat  decision  support? 

Task  Demands.  Because  the  command 
and  control  environment  is  complex, 
dynamic,  adversarial,  uncertain,  and  time 
critical,  it  poses  special  difficulties  and 
challenges  for  interface  design.  Complex 
information  processing  and  the  dynamic 
environment  make  extreme  cognitive 
demands  on  command  and  the  combat 
team.  An  intelligent  adversary  adds  yet 
other  dimensions  of  deception  and 
countermeasures  as  enemy  intentions  must 
be  recognized  in  the  midst  of  uncertain, 
unreliable,  and  often  contradictory  data. 
Further,  combat  is  a  "high  stakes" 
encounter  in  which  operator  errors  may 
weil  be  catastrophic  and  time  is  critical. 

In  contrast  to  more  passive  applications, 
preventing  errors  is  greatly  preferred  to 
correcting  them  and  computer  response 
times  are  not  merely  matters  of  user 
convenience.  Time  constraints  (in  the 
form  of  windows  of  opportunity)  and  limits 
in  technology  thus  critically  determine  the 
division  of  labor  between  human  and 
computer. 

Hardware/Software  Options.  The 
decreasing  cost  of  computer  memory  and 
processing  increases  the  available  options 
for  combat  system  hardware  and  software. 
The  so-called  "expert  systems"  technology 
shows  much  promise.  An  expert  system  is 
a  computer  program  that  has  the 
knowledge,  procedures,  algorithms,  and 
heuristics  necessary  to  perform  a  task 
typically  done  by  hurnans.  The  most 
successful  expert  systems  work  In  well- 
defined,  manageable  domains. 
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Typically,  expert  systems  act 
independently  (e.g.,  automatic  detectors, 
trackers,  and  classification  systems) 
with  the  user  primarily  monitoring  and 
evaluating  the  output.  Human  factors 
research  for  such  expert  systems  deals 
with  traditional  interface  issues  such  as 
input/output  modes,  the  use  of  color, 
menu  design,  legibility  and  display 
formats,  and  the  use  of  new  tools  such  as 
display  windows  and  graphic  pointing 
devices  (the  "mouse"). 


Further  development  of  these 
"independent"  expert  systems  will 
probably  benefit  Navy  command  and 
control  systems  marginally.  What  is 
needed  is  better  integration  of  the  user 
and  the  computer  so  that  both  partners 
truly  collaborate  in  the  decision  making 
process.  Although  traditional  human 
factors  concerns  are  important  in  expert 
systems,  the  most  fundamental  human 
factors  issue  is  to  design  the  interface  so 
that  it  promotes  human-computer 
problem  solving.  Unfortunately,  military 
systems  seldom  use  collaborative 
designs.  However,  there  is  a  great  need 
for  them  in  present  and  future  combat 
systems. 


Model  of  User.  The  operator/ 
decision  maker  in  combat  systems  is 
indeed  resourceful,  skillful,  and 
knowledgeable.  However,  the  typical 
design  seems  to  assume  that  the  user  is 
always  in  control  (i.e.,  aware  of  all 
action  choices  and  system  assets).  In 
fact,  memory  and  attention  limits,  the 
ability  to  integrate  information,  multiple 
task  demands,  cognitive  biases,  stress, 
and  incomplete  knowledge  of  the 
system's  support  capabilities  constrain 
the  user.  An  appropriate  model  of  the 
user  would  incorporate  these  realities 
and  lead  to  the  design  of  more 
intelligent  support  systems. 


Model  of  the  System.  The  design 
metaphor  of  the  computer  as  a  staff 
assistant  captures  the  kind  of  active 
collaborative  support  that  intelligent 
combat  decision  aids  require.  Such  an 
assistant  must  have  extensive  knowledge  of 
the  environment,  as  well  as  the  procedures, 
strategies,  and  functions  that  complement 
the  user's  cognitive  capabilities.  A  balance 
of  initiative,  cooperation,  and  faithful 
execution  is  essential. 

Some  plausible  roles  for  an  intelligent 
command  and  control  assistant  are  to; 

1.  Provide  easy  access  to  tools. 

2.  Automate  routine  functions. 

3.  Execute  specific  subtasks  on 
demand. 

4.  Inform  the  user  of  the  status  of 
tasks. 

5.  Remind  the  user  to  perform 
certain  tasks. 

6.  Advise  the  user  in  selecting 
alternative  actions. 

7.  Monitor  progress  toward  the  goal. 

8.  Anticipate  requests  to  display  or 
process  data. 

9.  Test  hypotheses. 

10.  Explain  its  behavior. 

Additional,  and  perhaps  more  intelligent 
assistance  must  be  defined  in  the  context 
of  specific  applications,  but,  clearly,  a 
passive  interface  will  not  suffice  for  this 
type  of  collaboration.  Instead,  advanced 
user-computer  communication  concepts 
are  required  to  promote  the  full 
integration  of  the  computerized  staff 
assistant  for  combat  systems  design. 
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Research  and  Development  Test  Bed 

Specifying  intelligent  interfaces  for 
collaborative  expert  systems  is  far  from 
complete  and  is  an  active  research  topic 
in  the  human  factors,  cognitive  science, 
and  artificial  intelligence  disciplines. 

Our  research  explores  advanced  concepts 
of  user-computer  collaboration  in  a 
laboratory  test  bed,  called  intelligent 
tactical  assistant  (IT  A).  IT  A  Is  a 
prototype  aid  for  identifying  ship 
contacts  based  on  electronic  and 
acoustic  emissions  and  intelligence 
reports.  Its  data  base  contains  sensor 
and  weapons  information  and  other 
descriptive  data  for  28  classes  of  Soviet 
ships.  IT  A  provides  tools  for  searching 
the  data  base  quickly  and  integrating 
new  reports  with  current  information.  It 
was  initially  implemented  in  the  LISP 
language  on  a  VAX  computer.  In  F  Y85, 
IT  A  was  redesigned  (using  C  and  LISP 
code)  for  the  NPRDC  System  Simulation 
Facility's  MASSCOMP  computer.  This 
version  has  a  color  graphics  display  to 
facilitate  data  integration  and  a 
mouse/window  interface  for  ready 
access  and  manipulating  of  the  data 
base. 


PLANS 

We  plan  to  incorporate  contextual 
knowledge  in  the  form  of  system  goals 
and  the  history  of  the  interaction  into 
ITA.  1TA  needs  this  contextual  knowl¬ 
edge  about  means  and  ends  in  the 


contact  identification  problem  to  act  as  a 
true  assistant.  However,  each  partner  in 
collaborative  expert  systems  must  have  a 
model  of  the  other's  objectives  and 
methods,  as  well  as  effective  rr^ans  of 
communicating  status,  questions,  answers, 
etc.  Thus,  there  are  formidable  human 
factors  research  issues  in  collaborative 
expert  systems  for  combat. 


The  planned  research  will  focus  on  the 
dynamics  of  an  interface  that  can  support 
both  independent  and  collaborative  efforts 
of  the  user  and  the  computer.  This  focus 
suggests  that  multiple  working  surfaces 
will  be  required:  A  display  of  the  current 
solution  that  reflects  the  team's  progress, 
a  display  for  the  user  to  explore  possible 
alternatives,  and  a  display  for  the  assistant 
to  inform  the  user  about  ongoing  analyses. 
Various  questions  then  arise  in  the 
collaborative  environment:  Under  what 
conditions  should  the  assistant  interrupt 
the  user?  What  are  the  best  forms  for 
user-assistant  dialogues?  How  much 
adaptation  and  flexibility  does  the  combat 
system  interface  require?  How  does  one 
evaluate  the  performance  of  the  new 
partnership? 


EXPECTED  PAYOFF 

The  findings  from  this  effort  relate 
directly  to  fleet  requirements  for 
increased  operability  of  combat  systems. 
The  results  are  expected  to  proceed  to 
advanced  development  within  three  years. 


DESIGNER:  AN  EXPERT  GRAPHICAL  DESIGN  CONSULTANT 


Louis  Weitzman 


Applying  new  technologies  in  artificial  intelligence  and  cognitive  science 
to  the  development  of  interactive  computer-based  training  systems  can 
provide  support  in  areas  where  the  developers  lack  expertise.  For  example , 
the  Graphics  Editor  developed  in  connection  with  Steamer  (a  training  system 
for  the  instruction  of  steam  propulsion),  allows  nonprogrammers  to  create 
dynamic,  inspectable  computer  diagrams  interactively.  The  research 
described  here  focuses  on  the  development  of  Designer,  an  interactive  visual 
design  consultant  for  users  of  this  editor.  Designer  will  aid  developers  by 
providing  graphic  expertise  during  the  construction  and  critiquing  of 
Steamer  diagrams.  This  expertise  includes  graphic  design  principles  as  well 
as  standards  of  presentation.  Designer  is  expected  to  improve  the  quality  of 
the  instruction  by  making  the  diagrams  more  consistent  and  visually  more 
effective. 


BACKGROUND 

The  development  of  interactive 
training  systems  and  human-computer 
interfaces  requires  expertise  in  a  number 
of  different  domains.  Currently  a 
developer  needs  to  be  knowledgeable 
about  the  substantive  domain  of  the 
application  as  well  as  computer 
programming,  instruction,  and  interface 
design.  Emerging  new  technologies  in 
artificial  intelligence  and  cognitive 
science  provide  opportunities  to  create 
computer-based  tools  for  providing 
support  in  domains  outside  of  the 
developers'  areas  of  expertise.  For 
example,  Steamer,  a  computer-based 
training  system  for  propulsion 
engineering,  provides  a  Graphics  Editor 
that  allows  nonprogrammers  to 
graphically  create  interactive  dynamic 
diagrams.  This  facility  has  allowed 
propulsion  engineering  instructors  to 


create  substantial  portions  of  this 
advanced  training  system.  They  have  used 
this  editor  to  create  over  100  diagrams 
that  serve  as  the  students'  interface  to  a 
steam  plant  simulation.  However,  users  of 
the  Steamer  Graphics  Editor  have  had 
difficulty  maintaining  drawing  conventions 
across  and  even  within  diagrams.  In 
addition,  important  graphic  design 
principles  are  often  violated. 


OBJECTIVE 

This  research  is  being  conducted  to 
provide  a  mechanism  for  users  of  the 
Graphics  Editor  to  maintain  diagram 
conventions  and  more  fully  exploit 
techniques  of  good  graphic  design  in  the 
construction  of  Steamer  diagrams.  More 
generally,  it  is  the  objective  of  this 
research  to  explore  the  use  of  explanatory 


Deilgner  U  •  tool  to  «ld  in  the  conitruction  ol  computer  Interfacet  uted  In  inttructlonal  training  iyttema.  It  provide* 
graphical  detlgn  expertlte  while  constructing  interactive  and  I  rupee  table  views.  The  left  screen  present*  the  view  while 
Louis  Weitzman  interacts  with  a  multl-paned  graphics  editor  window  on  the  right 


expert  systems  in  the  design  process. 
Designer  will  be  an  interactive  visual 
design  consultant  that  will  incorporate 
generic  graphic  design  principles  and 
drawing  standards  to  critique  and  aid  in 
the  construction  of  Steamer  diagrams. 
The  resulting  diagrams  will  be  visually 
more  effective  and  stylistically  more 
consistent.  This  system  will  increase  the 
productivity  of  the  developers  while 
enhancing  the  effectiveness  of  the 
instruction. 

Another  goal  of  this  project  is  to 
create  a  general  tool  for  assisting  the 
design  process.  By  providing  an 
independent  framework  for  critiquing, 
explaining,  and  exploring  design 


alternatives,  the  domain  of  the  critique 
can  be  changed  without  affecting  the 
structure  of  the  critiquing  mechanism 
itself.  Even  though  these  designs  will  be 
analyzed  in  the  domain  of  the  graphic  arts, 
the  same  framework  can  be  employed  to 
provide  critiques  in  other  domains  (e.g., 
circuit  design,  architectural  design). 

In  addition  to  merely  describing  design 
alternatives,  the  system  will  provide  a 
facility  for  exploring  the  design  space;  that 
is,  to  explain  the  advantages  and  disad¬ 
vantages  of  the  different  design  solutions 
through  interactive  dialogue  and  construc¬ 
tive  examples,  the  system  will  tutor 
students  in  the  principles  of  design  for  the 
given  domain. 


APPROACH 


Designer  consists  of  three 
interrelated  processes,  an  Analyzer,  a 
Critiquer,  and  a  Synthesizer,  coupled  to 
a  domain  dependent  knowledge  base. 

This  knowledge  base  consists  of  design 
elements  and  relationships,  techniques 
for  their  identification,  and  the 
constraints  used  in  distinguishing  good 
design  from  bad.  The  Analyzer  first 
parses  the  design  based  on  the  elements 
and  relationships  of  the  given  domain. 

1  le  Critiquer  uses  this  information  to 
indicate  where  the  current  design  fails 
to  conform  to  principles  of  good  design 
or  predetermined  guidelines.  Finally, 
based  on  searches  of  the  design  space, 
the  Synthesizer  process  suggests 
alternative  modifications  to  the  current 
state  of  the  design.  The  alternatives  are 
explained  in  terms  of  their  potential 
contribution  to  the  final  solution.  The 
separation  of  these  three  processes  from 
the  knowledge  base  provides  the  desired 
independence  and  modularity  of  the 
system. 


Domain-based  design  constraints  are 
the  basis  of  the  critiquing  process. 
Constraints  within  Designer  consist  of 
both  basic  graphic  design  principles 
important  in  the  construction  of  two- 
dimensional  diagrams  and  diagram  stand¬ 
ards  that  have  been  adopted  for  Steamer 
diagrams  (e.g.f  format  of  titles  and  pipe 
width  standards).  The  combination  cf 
principles  and  standards  create  a  context 
or  style  within  which  the  design  critique 
and  subsequent  modifications  take  place. 
By  modifying  the  style  within  which  a 
critique  is  made,  one  can  ultimately 
affect  the  form  of  the  final  design.  It 
thus  becomes  possible  under  this 
paradigm  to  request  multiple  critiques, 
each  based  on  a  different  style.  This 
paradigm  is  especially  powerful  for 
designs  that  may  be  presented  in 
different  media,  each  with  different 
constraints  that  need  to  be  considered. 


For  example,  a  style  appropriate  for  a 
high-resolution  color  display  on  a  computer 
screen  may  not  be  appropriate  for  a  black 
and  white  presentation  on  hard  copy  where 
features  are  not  as  clearly  distinguishable. 

PROGRESS 

An  initial  implementation  of  Designer 
is  under  development.  A  functioning 
Analyzer  and  Critiquer  used  on  existing 
Steamer  diagrams  have  provided  useful 
feedback.  It  was  very  encouraging  that, 
even  in  the  diagrams  that  we  had  thought 
were  carefully  crafted,  this  early  version 
of  the  system  was  able  to  note 
inconsistencies  and  suggest  improvements. 

Progress  has  been  made  in  identifying 
the  basic  elements,  relationships,  and 
principles  of  two-dimensional  graphic 
design.  This  knowledge  has  been  incor¬ 
porated  in  the  construction  of  Designer's 
processes.  The  Analyzer  first  evaluates 
the  different  desigr  elements  in  terms  of 
their  size,  shape,  color,  and  location  and 
then  identifies  the  relationships  that  are  in 
the  knowledge  base.  These  relationships 
include  similarity,  proximity,  grouping,  and 
repetition.  As  new  relationships  are 
identified,  they  can  easily  be  added  into 
the  analysis  process  to  enhance  this  phase 
of  the  system.  Various  techniques  exist  to 
inspect  interactively  the  elements  and 
relationships  identified  within  the  design. 

The  Critiquer  locates  examples  and 
violations  of  the  design  constraints 
provided  in  the  current  style  and  creates 
the  critique  comments.  These  critique 
comments  include  descriptions  and  justifi¬ 
cations  based  on  the  graphic  constraint 
from  which  they  were  derived.  Since  the 
critiquer  works  within  the  context  of  a 
current  style,  there  are  facilities  to  help 
define  graphic  constraints  and  maintain 
styles.  A  preliminary  graphic  constraint 
language  allows  the  creation  of  new  con¬ 
straints,  while  a  style  editor  has  been 
developed  to  create,  maintain,  and  switch 
between  styles. 
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Figure  1  shows  Designer's  top  level 
user  interface.  On  the  right  side  are 
four  windows  with  commands  to  access 
diagrams:  describe  the  current  state  of 
a  diagram  critique;  analyze,  critique, 
and  synthesize  a  design;  and  maintain 
and  edit  styles.  Status  windows  attached 
to  these  four  panes  keep  track  of  the 
current  diagram,  the  number  of  elements 
and  relationships  analyzed,  the  state  of 
the  critique,  and  the  current  style, 
respectively.  The  middle  pane  lists 
examples  (+)  and  violations  (-)  of  the 
current  style's  constraints  with  a  more 
detailed  description  of  one  of  the 
violations  listed  in  the  larger  window 
pane  on  the  left.  Figure  1  also  shows  a 
pop-up  menu  that  is  available  to  edit  the 
current  style,  New  Style  1. 


PLANS 

Much  of  the  current  system  is  a 
framework  built  for  exploring  the  research 
issues  in  a  working  system.  This 
preliminary  system  is  performing  well,  but, 
in  many  areas,  functionality  has  yet  to  be 
developed  or  existing  facilities  need  to  be 
enhanced. 

Expansion  of  the  knowledge  base  of 
graphic  design  principles.  With  an 
expanded  domain  knowledge  base,  a  more 
thorough  and  detailed  analysis  and  critique 
can  be  provided  and  a  larger  design  space 
of  alternatives  can  be  explored.  Better 
techniques  for  parsing  the  diagram  in 
terms  of  these  principles  should  produce  a 
less  sensitive  and  more  stable  critique. 
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Enhancement  of  the  explanation  and 
description  facilities.  Currently  the 
explanation  facility  is  very  limited.  The 
system  should  be  able  to  generate  better 
explanations  of  the  existing  state.  A 
history  mechanism  might  also  help  in  the 
description  and  explanation  of  the 
diagram's  current  state.  Making  the 
analysis,  critique,  and  synthesis 
processes  easier  to  inspect  should  also 
assist  the  user  to  improve  the  diagram. 
Only  a  limited  number  of  techniques  are 
currently  employed  in  this  area  such  as 
pointing  to  an  element  and  having  it 
describe  itself,  graphically  indicating  a 
design  relationship,  or  examining  a 
critique  comment  by  visually  marking 
the  elements  involved. 

Exploration  of  the  design  space. 

The  least  developed  segment  of  the 
system  is  exploring  the  design  space. 

The  existing  early  prototype  needs  much 
more  development  in  order  to  aid  the 
user  ir.  considering  alternatives. 


Creating  new  and  enhancing  existing 
system  tools.  Building  tools  to  aid  in  the 
construction  and  maintenance  of  a  system 
can  be  quite  helpful.  In  Designer,  a  Style 
Editor  manages  the  various  styles  used  in 
the  critiquing  process.  An  enhanced  Style 
Editor  would  be  helpful  in  managing  the 
system's  existing  styles,  as  well  as  in 
creating  new  ones.  A  Constraint  Editor, 
built  on  a  graphical  constraint  language,  is 
planned  to  facilitate  the  construction  and 
maintenance  of  new  principles  and 
standards.  Another  powerful  tool  to  be 
incorporated  in  the  system  is  a  Truth 
Maintenance  System  (TMS),  which 
maintains  a  set  of  assumptions  within  an 
environment.  At  this  point,  how  to  utilize 
a  TMS  within  Designer  has  not  been  clearly 
defined,  but  it  should  be  helpful  in 
identifying  contradictions  within  a  design 
or  explaining  why  it  is  valid. 
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APPENDIX  A 

Transitioned  and  Completed  Independent  Research/ 
Independent  Exploratory  Development 
Work  Units 
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Table  A-l 

Independent  Research  Work  Units  for  FYE5  and  FY86  (PE  61 152N) 


Principal 

Internal 

Telephone 
(619)  225- 

FY 

Funding 

_J$K) 

Work  Unit 

Title 

Investigator 

Code 

or  AV  933- 

85 

86 

ZROOO-Ol  042-04 

.024 

Models  for  Calibrating 
Multiple-choice  Items 

Mr.  3.  B.  Sympson 

62 

6153 

69a 

40 

.025 

Approaches  to  Multiple 
Objective  Assignment 

Dr.  T.  T.  Liang 

61 

2371 

0 

90 

ZROOO-Ol  042-06 

.014 

Enhancing  Understanding 
of  Electric  Circuits 

Dr.  W.  E.  Montague 

05 

7100 

30 

0b 

.025 

Cognitive  and  Affective 
Processing 

Dr.  B.  A.  McDonald 

51 

6434 

15 

0C 

.027 

Cognitive  Factors  in 
Learning  and  Retention 

Dr.  P.  3.  Konoske/ 
Dr.  3.  A.  Ellis 

72 

51 

2191 

6434 

10 

0d 

.028 

Cognitive  Storage 
Mechanisms 

Dr.  M.  S.  Baker 

52 

6955 

14 

0e 

.029 

Performance  on  Computer- 
based  Cognitive  T asks 

Dr.  P-A.  Federico 

51 

6434 

0 

60 

.030 

Cognitive  Robotics 

Dr.  F.  R.  Chang 

51 

6434 

0 

50 

ZROOO-Ol  042-0S 

.030 

Positively  Accelerating 
Sharing  Rates 

Dr.  D.  M.  Nebeker 

72 

6935 

251 

0d 

.031 

Military  Leadership 

Dr.  L.  E.  Atwater 

04 

6122 

34 

0C 

.032 

Management  Practices 
and  Organizational 
Performance 

Dr.  M.  C.  Butler/ 

Dr.  K.  S.  Crawford 

72 

6935 

35 

0C 

ZROOO-Ol  042-09 

.020 

Analysis  of  Cognition 
in  Natural  Settings 

Dr.  E.  L.  Hutchins 

501 

452- 

6771 

80 

50 

.027 

Expert  Systems  for 

Fault  Diagnosis 

Dr.  D.  B.  Malkoff 

71 

6617 

38 

0b 

.028 

Brain  Mechanisms  for 
Human  Color  Vision 

Dr.  L.  3.  T rejo 

71 

6617 

0 

50 

ZROOO-Ol  042-99 

To  be  distributed 

0 

32 

350 

372 

aCost  shared  with  Center  6.2  funding  ($20K  from  PE  62763N). 
^Independent  Exploratory  funding  for  FY86. 


r 

Concluded. 

^Transitioned. 

terminated  due  to  Center  priorities. 

fCost  shared  with  Center  6.2  funding  ($50K  from  PE  62763N1 
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Table  A-2 


Independent  Exploratory  Development 
Work  Units  for  FY85  and  FY86 
(PE  62766N) 


Work 

Unit 

Title 

Principal  Internal 

Investigator  Code 

T  elephone 
(619)  225- 
or  AV  933- 

FY 

Funding 

($*>... 

85  86 

ZF66-511 

.013 

Multiple  Criterion  Opti¬ 
mization  Techniques 

Dr.  T.  T.  Liang 

61 

2371 

15 

0“ 

ZF66-512 

.015 

Interfaces  for  Combat 
Decision  Support 

Dr.  F.  L.  Greitzer 

71 

2081 

90 

0 

.016 

Impact  of  Statistical 
Process  Control 

Dr.  L.  A.  Broedling 

52 

6522 

10 

ob 

.017 

Development  of 

Graphic  Design  Aids 

Mr.  L.  M.  Weitzman 

501 

7100 

50 

50 

.018 

Expert  Systems  for 

Fault  Diagnosis 

Dr.  D.  B.  Malkoff 

71 

6617 

oc 

100 

.019 

Changes  in  Cognitive 
Structures  T rainingd 

Dr.  W.  E.  Montague 

05 

7100 

0C 

165 

60 

210 

transitioned. 


^Concluded. 

C1R  funding  for  FY85. 

^Previously  titled  "Enhancing  Understanding  of  Electric  Circuits." 
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Table  A-3 


Independent  Research  and  Independent  Exploratory  Development 
Work  Units  Completed  At  the  End  of  FY85 


Work  Unit 

Title 

Reason  for 
Termination 

Independent  Research  Work  Units 

ZR000-01  042-06 

.014 

Enhancing  Understanding  of 
Electric  Circuits 

IED  funding 
(see  Table  A-2). 

.025 

Cognitive  and  Affective 

Processing 

Concluded. 

.027 

Cognitive  Factors  in  Learning 
and  Retention 

Transitioned 
(see  Table  A-4). 

.028 

Cognitive  Storage  Mechanisms 

Terminated  due  to 
Center  priorities. 

042-08 

.030 

Positively  Accelerated  Sharing 
Rates 

Transitioned 
(see  Table  A-4). 

.031 

Military  Leadership 

Concluded. 

.032 

Management  Practices  and 
Organizational  Performance 

Concluded. 

042-09 

.02  7 

Expert  Systems  lor  Fault 

Diagnosis 

Transitioned  to  IED 
(Table  A-2). 

Independent  Exploratory  Development  Work  Units 

F66-511  ZF66-51 1 

.013 

Multiple  Criterion  Optimization 
Techniques 

Transitioned 
(see  Table  A-4). 

ZF66-512 

.015 

Interfaces  for  Combat  Decision 
Support 

Transitioned 
(see  Table  A-4). 

.016 

Impact  of  Statistical  Process 

Concluded. 

Control 


Table  A -4 


Transitioned  Independent  Research  and  Independent 
Exploratory  Development  Work  Units  (FY85) 


IR/IED  Title 

New  Title 

Sponsor 

Program 

Element 

FY86 

Expected 

Funding 

($K) 

Cognitive  Factors 

1.  Functional  Context 

CNO  (OP-G1) 

63720N 

500 

in  Learning  and 

Training 

Retention 

2.  Classroom  Instruc- 

ONT 

62763N 

315 

tional  Technologies 

Positively  Acceler- 

Improving  Individual  and 

ONT 

62763N 

295 

ated  Sharing  Rates 

Unit  Productivity 

Independent  Exploratory  Development  Work  Units 

Multiple  Criterion 

1.  Career  Management 

ONT 

62763N 

340 

Optimization  Tech- 

Planning 

niques 

2.  Personnel  Assign- 

CNO  (OP-01) 

63707N 

430 

ment  Systems 

Interfaces  for 

Decision  Aiding  in 

ONT 

62757N 

230 

Combat  Decision 

Combat  Systems 

Support 

Note.  CNO  =  Chief  of  Naval  Operations;  ONT  =  Office  of  Naval  Technology, 
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INDEPENDENT  RESEARCH/INDEPENDENT  EXPLORATORY  DEVELOPMENT 
RESEARCHERS'  AWARDS  AND  HONORS 


Atwater,  L.  E. 

Judge  at  International  Science  and  Engineering  Fair,  Shreveport,  LA  (May  1985). 
Broedling,  L.  A. 

Invited  presentation  on  the  subject  of  quality  management  to  the  Total  Quality 
Control  Council  of  Rogers  Corp  (March  1985). 

Invited  presentation  on  the  subject  of  quality  management  to  the  California  Society 
of  Professional  Engineers  Annual  Convention  (June  1985). 

Crawford,  K.  S. 

Recipient  of  NAVPERSRANDCEN's  First  Annual  Professional  Publications  Award  for 
the  most  outstanding  article  published  in  a  scientific  journal  (January  1985). 

Invited  presentation,  "Using  Gain  Sharing  Programs  to  Improve  Productivity  in  the 
Federal  Government,"  at  23rd  Biennial  Federal  Personnel  Management  Conference,  San 
Francisco  (November  1984). 

Received  Performance  Management  Recognition  System  Award  (1984-1985). 

Ellis,  J.  A. 

Member  of  the  Editorial  Board  for  the  Journal  of  Educational  Psychology. 

Invited  workshop  presentation,  "Instructional  Quality  Control  and  Test  Development," 
for  West  Coast  Shipyard  Training  Developers,  Received  a  letter  of  commendation  from 
the  Naval  Sea  Systems  Command. 

Greitzer,  F.  L. 

Invited  presentation,  "Intelligent  Tactical  Assistant,"  Technical  Cooperation  Panel, 
(July  1985). 

Hutchins,  E.  L. 

Awarded  John  D.  and  Catherine  T.  MacArthur  Foundation  Fellowship  for  1985-1990. 

Invited  presentation,  "Color  Nomenclature  Research,"  Department  of  Anthropology, 
departmental  colloquium,  Duke  University  (January  1985). 

Invited  presentation,  "Direct  Manipulation  Interfaces,"  departmental  colloquium, 
School  of  Education,  University  of  California,  Berkeley  (March  1985). 
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Invited  presentation,  "Direct  Manipulation  Interfaces,"  presented  at  ONR  Contractors 
Conference,  University  of  Washington,  Seattle,  WA  (May  1985). 

Invited  presentation,  "Maneuvering  Board  Training  System,"  Operation  Specialist 
Advisory  Board  Meeting,  San  Die$,o,  CA  (May  1985). 

Invited  presentation,  "Future  Technologies  for  Training,"  Navy-wide  Technical 
Review  of  Artificial  Intelligence  Research  and  Development,  Washington,  DC  (September 
1985). 

Liang,  T.  T. 

Received  Performance  Management  Recognition  System  Award  (1984-1985). 

Malkoff,  D.  B. 

Invited  presentation,  "Recent  Progress  in  Introducing  Artificial  Intelligence  Into 
Brain  Analysis,"  UC  San  Diego  School  of  Medicine,  Department  of  Neurosciences,  San 
Diego,  CA  (June  1985). 

Invited  participant  at  Brain  Mapping  Workshop,  sponsored  by  the  Army  Medical 
Research  and  Development  Command,  Texax  A&M  University  (August  1985). 

Member  of  the  American  Academy  of  Neurology,  American  Medical  EEG  Association, 
American  Association  for  Medical  Systems  and  Informatics,  and  the  Society  for  Neuro¬ 
science. 

McDonald,  B.  A 

Received  letter  of  commendation  from  the  Fleet  Combat  Training  Center,  Pacific 
for  major  contributions  in  the  areas  of  electronic  warfare  and  electronic  countermeasure 
training  over  a  3-year  period  and  for  work  in  cognition  and  emotion  as  related  to  Tactical 
Decision  Making. 

Montague,  W.  E. 

Member  of  Editorial  Boards  for  the  Journal  of  Experimental  Aging  Research. 
Educational  Psychologist,  Journal  of  Educational  Psychology,  American  Educational 
Research  Journal.  Educational  Evaluation  and  Policy  Analysis,  and  Training  Technology. 

Proposal  reviewer  for  Office  of  Naval  Research  and  National  Science  Foundation. 

Invited  presentation,  "Computer-based  Training  and  Instructional  Quality,"  Gould 
Association. 

Invited  presentation,  "Where's  the  Intelligence  in  Intelligent  CBI7",  National  Society 
for  Performance  and  Instruction,  San  Diego  Chapter. 

Invited  presentation,  "Computer-based  Instruction,"  Air  Force  Technology  in  Training 
and  Education  Conference. 
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Invited  presentation,  ’’Implementation  of  Instructional  Technology,"  Office  of  Naval 
Research  Conference. 

Invited  presentation,  "Workshop  on  Intelligent  Tutoring  Systems,"  Courseware,  Inc. 

American  Psychological  Association,  Divisions  21  and  14  jointly  sponsored  invited 
workshop.  Applications  of  Technology  in  Training,  Los  Angeles,  CA  (August  1985). 

Received  letter  of  commendation  for  USMC  Individual  Training  Standards  Program 
Review  (March  1985). 

Member  of  Chief  of  Naval  Education  and  Training  committees  on  embedded  training 
and  rate  training. 

President,  Human  Factors  Society,  San  Diego  Chapter. 

Nebeker,  D.  M. 

Invited  to  press  conference  sponsored  by  Minolta,  Corp.  in  New  York,  "Employee  gain 
sharing  for  office  professionals:  Are  we  missing  an  opportunity?"  (April  1985). 

Member  of  Editorial  Board  of  Organizational  Behavior  and  Human  Decision 
Processes. 

Invited  member  of  Society  for  Organizational  Behavior,  which  limits  itself  to  60 
members  worldwide. 

Invited  presentation  to  Workforce  Effectiveness  Division  Head  of  Office  of  Personnel 
Management  regarding  research  at  the  NAVPERSRANDCEN  Organizational  Systems 
Simulation  Laboratory. 

Sympson,  J.  B. 

Invited  presentation,  "Stochastic  Modeling  in  Psychological  Test  Theory,"  Operations 
Research  Department,  Naval  Postgraduate  School,  Monterey,  CA  (February  1985). 

Invited  review  of  Applications  of  Item  Response  Theory  published  in  Applied 
Psychological  Measurement  Journal  (Vol.  8,  No.  4,  pp.  467-470). 


Manuscript  reviewer  for  Psychometrika  and  Journal  of  Educational  Measurement. 
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INDEPENDENT  RESEARCH  PRESENTATIONS 


Konoske,  P.  3.,  &  Ellis,  3.  A. 

Effects  of  Learning  and  Retention  of  Procedural  Tasks.  Paper  presented  to  the 
American  Educational  Research  Association,  Chicago,  IL  (April  1985). 

Malkoff,  D.  B. 

Spoke  on  his  1R  research  project,  Expert  Systems  for  Fault  Diagnosis,  at  the  U.S. 
Naval  Academy,  Naval  Research  Laboratory,  Office  of  Naval  Research,  and  Naval  Sea 
Systems  Command  (April,  May  1985). 

Morrison,  R.  F. 

Implications  for  career  choice  and  planning.  Invited  address  for  Division  14, 
American  Psychological  Association  Annual  Meeting,  Los  Angeles  (August  1985). 

Sympson,  3.  B. 

"Principal  Component  Analysis  of  Polychotomous  Item  Responses,"  and  "A  Family  of 
Distribution  Functions"  presented  at  the  Office  of  Naval  Research  conference  on  Model- 
based  Psychological  Measurement,  Princeton,  N3  (October  1984). 


INDEPENDENT  EXPLORATORY  DEVELOPMENT  PRESENTATIONS 


Chang,  F.  R. 

Development  of  computer-based  writing  aids.  Paper  presented  at  the  annual  meeting 
of  the  American  Educational  Research  Association,  Chicago,  IL  (April  1985). 

Writing  text  for  many  readers:  A  role  for  technology.  Paper  presented  at  the 
international  conference  on  The  Future  of  Literacy  in  a  Changing  World:  Synthesis  From 
the  Industrialized  and  Developing  Nations.  University  of  Pennsylvania,  Philadelphia,  PA 
(May  1985). 

Liang,  T.  T. 

A  method  to  generate  policy  goals  for  the  Navy's  assignment  problem.  Presented  to 
the  12th  International  Symposium  on  Mathematical  Programming  at  the  Massachusetts 
Institute  of  Technology,  Cambridge,  MA  (August  1985). 
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INDEPENDENT  RESEARCH  PUBLICATIONS 


Federico,  P-A  (1985).  Cognitive  complexity  and  cerebral  sensory  interaction.  Personality 
and  Individual  Differences,  6(2).  253-261. 

Federico,  P-A  (1985).  Individual  differences  in  concept  learning  and  brain  event-related 
potentials.  Personality  and  Individual  Differences,  6(2),  243-252. 


INDEPENDENT  EXPLORATORY  DEVELOPMENT  PUBLICATIONS 

Liang,  T.  T.,  &  Shao-3u,  L.  (December  1985).  A  systems  approach  to  integrate  manpower 
planning  and  operation.  Socio-Economic  Planning  Sciences.  _19(4). 

Liang,  T.  T,  &  Thompson,  T.  3.  (In  press).  A  large  scale  personnel  assignment  model  for 
the  Navy.  Decision  Sciences. 
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